Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:14:28.645943 21737 test_util.cc:268] Using random seed: 949900306
I20240502 14:14:28.656838 21737 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:14:28.656909 21737 ts_itest-base.cc:113] --------------
I20240502 14:14:28.656960 21737 ts_itest-base.cc:114] 6 tablet servers
I20240502 14:14:28.657020 21737 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:14:28.657090 21737 ts_itest-base.cc:116] --------------
2024-05-02T14:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:14:28Z Disabled control of system clock
I20240502 14:14:28.680131 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:41863
--webserver_interface=127.21.58.126
--webserver_port=0
--builtin_ntp_servers=127.21.58.84:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.58.126:41863
--location_mapping_cmd=/tmp/dist-test-task3cgK4A/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:14:28.988279 21749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:28.988634 21749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:28.988756 21749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:28.995587 21749 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:28.995714 21749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:14:28.995784 21749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:28.995858 21749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:14:28.995918 21749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:14:29.004078 21749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task3cgK4A/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.21.58.126:41863
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:41863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.58.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:29.005139 21749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:29.006652 21749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:29.016619 21754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:29.017078 21757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:29.017478 21755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:29.018015 21749 server_base.cc:1034] running on GCE node
I20240502 14:14:29.018606 21749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:29.019825 21749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:29.021013 21749 hybrid_clock.cc:648] HybridClock initialized: now 1714659269020977 us; error 25 us; skew 500 ppm
I20240502 14:14:29.021322 21749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:29.023439 21749 webserver.cc:460] Webserver started at http://127.21.58.126:42581/ using document root <none> and password file <none>
I20240502 14:14:29.024106 21749 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:29.024212 21749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:29.024488 21749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:29.026759 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "89f88a26537f4da7a7cf931ff11e658c"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.027474 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "89f88a26537f4da7a7cf931ff11e658c"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.032150 21749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:14:29.035176 21763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:29.036249 21749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:29.036397 21749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
uuid: "89f88a26537f4da7a7cf931ff11e658c"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.036577 21749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:29.048818 21749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:29.049424 21749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:29.049655 21749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:29.066115 21749 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.126:41863
I20240502 14:14:29.066135 21814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.126:41863 every 8 connection(s)
I20240502 14:14:29.067299 21749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:14:29.070683 21815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.073114 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 21749
I20240502 14:14:29.073339 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:14:29.079401 21815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c: Bootstrap starting.
I20240502 14:14:29.082121 21815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.083042 21815 log.cc:824] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:29.085232 21815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c: No bootstrap required, opened a new log
I20240502 14:14:29.093597 21815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f88a26537f4da7a7cf931ff11e658c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 41863 } }
I20240502 14:14:29.093925 21815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.094018 21815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f88a26537f4da7a7cf931ff11e658c, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.094635 21815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f88a26537f4da7a7cf931ff11e658c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 41863 } }
I20240502 14:14:29.094902 21815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:29.095052 21815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:29.095247 21815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.098078 21815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f88a26537f4da7a7cf931ff11e658c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 41863 } }
I20240502 14:14:29.098608 21815 leader_election.cc:302] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f88a26537f4da7a7cf931ff11e658c; no voters:
I20240502 14:14:29.099237 21815 leader_election.cc:288] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:29.099364 21820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:29.101329 21820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [term 1 LEADER]: Becoming Leader. State: Replica: 89f88a26537f4da7a7cf931ff11e658c, State: Running, Role: LEADER
I20240502 14:14:29.101325 21815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:29.101907 21820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f88a26537f4da7a7cf931ff11e658c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 41863 } }
I20240502 14:14:29.106107 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f88a26537f4da7a7cf931ff11e658c. Latest consensus state: current_term: 1 leader_uuid: "89f88a26537f4da7a7cf931ff11e658c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f88a26537f4da7a7cf931ff11e658c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 41863 } } }
I20240502 14:14:29.106473 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:29.106848 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f88a26537f4da7a7cf931ff11e658c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f88a26537f4da7a7cf931ff11e658c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 41863 } } }
I20240502 14:14:29.107149 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:29.112787 21829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:29.116786 21829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:29.124621 21829 catalog_manager.cc:1347] Generated new cluster ID: 102dde2d7e8d4160bc0810c2552242bf
I20240502 14:14:29.124722 21829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:29.150770 21829 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:29.151940 21829 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:29.161551 21829 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 89f88a26537f4da7a7cf931ff11e658c: Generated new TSK 0
I20240502 14:14:29.162253 21829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:29.178655 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.65:0
--local_ip_for_outbound_sockets=127.21.58.65
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--builtin_ntp_servers=127.21.58.84:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:14:29.507972 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:29.508212 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:29.508337 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:29.515256 21839 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:29.515358 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:29.515511 21839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.65
I20240502 14:14:29.523908 21839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.65
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:29.525066 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:29.526690 21839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:29.537482 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:29.537884 21844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:29.538379 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:29.538671 21839 server_base.cc:1034] running on GCE node
I20240502 14:14:29.539189 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:29.539999 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:29.541182 21839 hybrid_clock.cc:648] HybridClock initialized: now 1714659269541137 us; error 43 us; skew 500 ppm
I20240502 14:14:29.541460 21839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:29.543485 21839 webserver.cc:460] Webserver started at http://127.21.58.65:39957/ using document root <none> and password file <none>
I20240502 14:14:29.544137 21839 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:29.544240 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:29.544523 21839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:29.546826 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4b0a27c175a3442dac0cae3f5b505110"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.548058 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4b0a27c175a3442dac0cae3f5b505110"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.552542 21839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:14:29.555635 21853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:29.556753 21839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:29.556887 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4b0a27c175a3442dac0cae3f5b505110"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.557053 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:29.571659 21839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:29.572293 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:29.572537 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:29.573401 21839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:29.574673 21839 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:29.574759 21839 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:29.574858 21839 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:29.574925 21839 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:29.603549 21839 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.65:40171
I20240502 14:14:29.603610 21965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.65:40171 every 8 connection(s)
I20240502 14:14:29.604641 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:14:29.613711 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 21839
I20240502 14:14:29.613905 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:14:29.615979 21966 heartbeater.cc:347] Connected to a master server at 127.21.58.126:41863
I20240502 14:14:29.616243 21966 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:29.616983 21966 heartbeater.cc:511] Master 127.21.58.126:41863 requested a full tablet report, sending...
I20240502 14:14:29.617206 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.66:0
--local_ip_for_outbound_sockets=127.21.58.66
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--builtin_ntp_servers=127.21.58.84:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:29.659373 21779 ts_manager.cc:194] Registered new tserver with Master: 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:29.661357 21779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.65:40277
W20240502 14:14:29.925055 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:29.925318 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:29.925410 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:29.932407 21970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:29.932502 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:29.932603 21970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.66
I20240502 14:14:29.940857 21970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.66
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:29.941954 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:29.943506 21970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:29.954468 21977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:29.954802 21979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:29.955307 21976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:29.955377 21970 server_base.cc:1034] running on GCE node
I20240502 14:14:29.955929 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:29.956732 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:29.957916 21970 hybrid_clock.cc:648] HybridClock initialized: now 1714659269957881 us; error 33 us; skew 500 ppm
I20240502 14:14:29.958184 21970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:29.960137 21970 webserver.cc:460] Webserver started at http://127.21.58.66:33843/ using document root <none> and password file <none>
I20240502 14:14:29.960834 21970 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:29.960937 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:29.961225 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:29.963614 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fe26b209fc9842d28d31c6243c869061"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.964339 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fe26b209fc9842d28d31c6243c869061"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.968837 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:14:29.971720 21985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:29.972816 21970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:29.972963 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fe26b209fc9842d28d31c6243c869061"
format_stamp: "Formatted at 2024-05-02 14:14:29 on dist-test-slave-wtgm"
I20240502 14:14:29.973134 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:30.001358 21970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:30.002097 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:30.002338 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:30.003217 21970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:30.004539 21970 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:30.004629 21970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.004739 21970 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:30.004812 21970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.033354 21970 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.66:40657
I20240502 14:14:30.033411 22097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.66:40657 every 8 connection(s)
I20240502 14:14:30.034514 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:14:30.035358 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 21970
I20240502 14:14:30.035543 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:14:30.038977 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:0
--local_ip_for_outbound_sockets=127.21.58.67
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--builtin_ntp_servers=127.21.58.84:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:30.047130 22098 heartbeater.cc:347] Connected to a master server at 127.21.58.126:41863
I20240502 14:14:30.047390 22098 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:30.048097 22098 heartbeater.cc:511] Master 127.21.58.126:41863 requested a full tablet report, sending...
I20240502 14:14:30.104754 21779 ts_manager.cc:194] Registered new tserver with Master: fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:30.105654 21779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.66:34709
W20240502 14:14:30.358536 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:30.358801 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:30.358923 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:30.366168 22101 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:30.366287 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:30.366433 22101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:14:30.374825 22101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:30.376000 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:30.377566 22101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:30.388458 22107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:30.388924 22108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:30.389235 22110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:30.389469 22101 server_base.cc:1034] running on GCE node
I20240502 14:14:30.389979 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:30.390756 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:30.391939 22101 hybrid_clock.cc:648] HybridClock initialized: now 1714659270391897 us; error 39 us; skew 500 ppm
I20240502 14:14:30.392203 22101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:30.394197 22101 webserver.cc:460] Webserver started at http://127.21.58.67:36045/ using document root <none> and password file <none>
I20240502 14:14:30.394886 22101 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:30.394994 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:30.395309 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:30.397596 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
format_stamp: "Formatted at 2024-05-02 14:14:30 on dist-test-slave-wtgm"
I20240502 14:14:30.398285 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
format_stamp: "Formatted at 2024-05-02 14:14:30 on dist-test-slave-wtgm"
I20240502 14:14:30.402799 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 14:14:30.405885 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.407116 22101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:30.407258 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
format_stamp: "Formatted at 2024-05-02 14:14:30 on dist-test-slave-wtgm"
I20240502 14:14:30.407469 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:30.430140 22101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:30.430826 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:30.431095 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:30.431943 22101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:30.433262 22101 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:30.433357 22101 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.433468 22101 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:30.433540 22101 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.464125 22101 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:43989
I20240502 14:14:30.464179 22228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:43989 every 8 connection(s)
I20240502 14:14:30.465266 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:14:30.465813 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 22101
I20240502 14:14:30.466006 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:14:30.469434 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:0
--local_ip_for_outbound_sockets=127.21.58.68
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--builtin_ntp_servers=127.21.58.84:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:30.478698 22229 heartbeater.cc:347] Connected to a master server at 127.21.58.126:41863
I20240502 14:14:30.478960 22229 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:30.479776 22229 heartbeater.cc:511] Master 127.21.58.126:41863 requested a full tablet report, sending...
I20240502 14:14:30.519734 21779 ts_manager.cc:194] Registered new tserver with Master: 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:30.520903 21779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:49787
I20240502 14:14:30.664324 21966 heartbeater.cc:503] Master 127.21.58.126:41863 was elected leader, sending a full tablet report...
W20240502 14:14:30.781221 22232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:30.781457 22232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:30.781536 22232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:30.788556 22232 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:30.788676 22232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:30.788803 22232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:14:30.796878 22232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:30.797945 22232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:30.799576 22232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:30.810407 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:30.810454 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:30.810863 22239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:30.811184 22232 server_base.cc:1034] running on GCE node
I20240502 14:14:30.811668 22232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:30.812436 22232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:30.813644 22232 hybrid_clock.cc:648] HybridClock initialized: now 1714659270813605 us; error 29 us; skew 500 ppm
I20240502 14:14:30.813920 22232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:30.815979 22232 webserver.cc:460] Webserver started at http://127.21.58.68:33437/ using document root <none> and password file <none>
I20240502 14:14:30.816797 22232 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:30.816923 22232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:30.817250 22232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:30.819733 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "597f2b4f10754298a4a428ccd386c7d3"
format_stamp: "Formatted at 2024-05-02 14:14:30 on dist-test-slave-wtgm"
I20240502 14:14:30.820354 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "597f2b4f10754298a4a428ccd386c7d3"
format_stamp: "Formatted at 2024-05-02 14:14:30 on dist-test-slave-wtgm"
I20240502 14:14:30.824704 22232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:14:30.827497 22247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.828616 22232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:30.828771 22232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "597f2b4f10754298a4a428ccd386c7d3"
format_stamp: "Formatted at 2024-05-02 14:14:30 on dist-test-slave-wtgm"
I20240502 14:14:30.828930 22232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:30.849205 22232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:30.849931 22232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:30.850196 22232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:30.851066 22232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:30.852427 22232 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:30.852521 22232 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.852635 22232 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:30.852707 22232 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:30.882403 22232 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:33981
I20240502 14:14:30.882462 22359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:33981 every 8 connection(s)
I20240502 14:14:30.883639 22232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:14:30.885898 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 22232
I20240502 14:14:30.886071 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:14:30.889000 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:0
--local_ip_for_outbound_sockets=127.21.58.69
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--builtin_ntp_servers=127.21.58.84:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:30.896008 22360 heartbeater.cc:347] Connected to a master server at 127.21.58.126:41863
I20240502 14:14:30.896256 22360 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:30.896960 22360 heartbeater.cc:511] Master 127.21.58.126:41863 requested a full tablet report, sending...
I20240502 14:14:30.940200 21779 ts_manager.cc:194] Registered new tserver with Master: 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:30.941262 21779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:39323
I20240502 14:14:31.108457 22098 heartbeater.cc:503] Master 127.21.58.126:41863 was elected leader, sending a full tablet report...
W20240502 14:14:31.195369 22363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:31.195605 22363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:31.195708 22363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:31.202430 22363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:31.202549 22363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:31.202694 22363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:14:31.210812 22363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:31.211930 22363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:31.213431 22363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:31.223940 22370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:31.223949 22369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:31.224457 22372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:31.224606 22363 server_base.cc:1034] running on GCE node
I20240502 14:14:31.225118 22363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:31.225890 22363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:31.227100 22363 hybrid_clock.cc:648] HybridClock initialized: now 1714659271227036 us; error 36 us; skew 500 ppm
I20240502 14:14:31.227391 22363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:31.229408 22363 webserver.cc:460] Webserver started at http://127.21.58.69:45049/ using document root <none> and password file <none>
I20240502 14:14:31.230047 22363 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:31.230149 22363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:31.230412 22363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:31.232692 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "fe26ecc72bed46179803517c549c5673"
format_stamp: "Formatted at 2024-05-02 14:14:31 on dist-test-slave-wtgm"
I20240502 14:14:31.233424 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "fe26ecc72bed46179803517c549c5673"
format_stamp: "Formatted at 2024-05-02 14:14:31 on dist-test-slave-wtgm"
I20240502 14:14:31.238008 22363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240502 14:14:31.241073 22378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:31.242225 22363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:31.242367 22363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fe26ecc72bed46179803517c549c5673"
format_stamp: "Formatted at 2024-05-02 14:14:31 on dist-test-slave-wtgm"
I20240502 14:14:31.242545 22363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:31.266520 22363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:31.267314 22363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:31.267590 22363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:31.268414 22363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:31.269726 22363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:31.269821 22363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:31.269946 22363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:31.270015 22363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:31.299947 22363 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:42619
I20240502 14:14:31.300072 22490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:42619 every 8 connection(s)
I20240502 14:14:31.301218 22363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:14:31.304983 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 22363
I20240502 14:14:31.305202 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:14:31.308478 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.70:0
--local_ip_for_outbound_sockets=127.21.58.70
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--builtin_ntp_servers=127.21.58.84:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:31.313982 22491 heartbeater.cc:347] Connected to a master server at 127.21.58.126:41863
I20240502 14:14:31.314273 22491 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:31.315145 22491 heartbeater.cc:511] Master 127.21.58.126:41863 requested a full tablet report, sending...
I20240502 14:14:31.367997 21779 ts_manager.cc:194] Registered new tserver with Master: fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:31.368943 21779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:39305
I20240502 14:14:31.523334 22229 heartbeater.cc:503] Master 127.21.58.126:41863 was elected leader, sending a full tablet report...
W20240502 14:14:31.628787 22494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:31.629011 22494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:31.629107 22494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:31.635905 22494 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:31.636001 22494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:31.636096 22494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.70
I20240502 14:14:31.644753 22494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:41863
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.70
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:31.645740 22494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:31.647306 22494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:31.658520 22501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:31.658604 22500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:31.659018 22503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:31.659448 22494 server_base.cc:1034] running on GCE node
I20240502 14:14:31.659933 22494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:31.660781 22494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:31.661970 22494 hybrid_clock.cc:648] HybridClock initialized: now 1714659271661935 us; error 29 us; skew 500 ppm
I20240502 14:14:31.662256 22494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:31.664252 22494 webserver.cc:460] Webserver started at http://127.21.58.70:44297/ using document root <none> and password file <none>
I20240502 14:14:31.664917 22494 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:31.665020 22494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:31.665324 22494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:31.667906 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "491bfc422e6440749f54ffffdb717e3a"
format_stamp: "Formatted at 2024-05-02 14:14:31 on dist-test-slave-wtgm"
I20240502 14:14:31.668716 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "491bfc422e6440749f54ffffdb717e3a"
format_stamp: "Formatted at 2024-05-02 14:14:31 on dist-test-slave-wtgm"
I20240502 14:14:31.673214 22494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:14:31.676200 22509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:31.677326 22494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:31.677460 22494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "491bfc422e6440749f54ffffdb717e3a"
format_stamp: "Formatted at 2024-05-02 14:14:31 on dist-test-slave-wtgm"
I20240502 14:14:31.677626 22494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:31.692780 22494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:31.693447 22494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:31.693691 22494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:31.694512 22494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:31.695827 22494 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:31.695928 22494 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:31.696036 22494 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:31.696105 22494 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:31.725106 22494 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.70:40085
I20240502 14:14:31.725154 22621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.70:40085 every 8 connection(s)
I20240502 14:14:31.726205 22494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:14:31.733898 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 22494
I20240502 14:14:31.734066 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:14:31.737349 22622 heartbeater.cc:347] Connected to a master server at 127.21.58.126:41863
I20240502 14:14:31.737592 22622 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:31.738245 22622 heartbeater.cc:511] Master 127.21.58.126:41863 requested a full tablet report, sending...
I20240502 14:14:31.778676 21779 ts_manager.cc:194] Registered new tserver with Master: 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:31.779554 21779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.70:42279
I20240502 14:14:31.785979 21737 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:14:31.803848 21737 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:14:31.943819 22360 heartbeater.cc:503] Master 127.21.58.126:41863 was elected leader, sending a full tablet report...
I20240502 14:14:32.371727 22491 heartbeater.cc:503] Master 127.21.58.126:41863 was elected leader, sending a full tablet report...
I20240502 14:14:32.781963 22622 heartbeater.cc:503] Master 127.21.58.126:41863 was elected leader, sending a full tablet report...
I20240502 14:14:35.557148 21779 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:35.601841 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 38de82b0fbb54b8280debbea89a57432 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.603474 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38de82b0fbb54b8280debbea89a57432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.611454 22557 tablet_service.cc:1467] Processing CreateTablet for tablet e999c7d734334277b470ac0ebbee304f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.611454 22164 tablet_service.cc:1467] Processing CreateTablet for tablet e999c7d734334277b470ac0ebbee304f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.612823 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e999c7d734334277b470ac0ebbee304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.612823 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e999c7d734334277b470ac0ebbee304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.615240 22163 tablet_service.cc:1467] Processing CreateTablet for tablet b5c3ece6f2614827b2b5abd95228ac6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.615680 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c3ece6f2614827b2b5abd95228ac6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.619858 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 38de82b0fbb54b8280debbea89a57432 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.621132 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38de82b0fbb54b8280debbea89a57432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.624560 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 38de82b0fbb54b8280debbea89a57432 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.625198 22425 tablet_service.cc:1467] Processing CreateTablet for tablet b5c3ece6f2614827b2b5abd95228ac6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.625569 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c3ece6f2614827b2b5abd95228ac6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.626870 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38de82b0fbb54b8280debbea89a57432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.626770 21900 tablet_service.cc:1467] Processing CreateTablet for tablet b5c3ece6f2614827b2b5abd95228ac6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.628139 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c3ece6f2614827b2b5abd95228ac6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.631145 21901 tablet_service.cc:1467] Processing CreateTablet for tablet e999c7d734334277b470ac0ebbee304f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6382a94b0ee442a7bf0a2c1d3bce704a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.631539 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e999c7d734334277b470ac0ebbee304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.651573 22649 tablet_bootstrap.cc:492] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:35.655031 22648 tablet_bootstrap.cc:492] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:35.655632 22647 tablet_bootstrap.cc:492] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:35.658136 22648 tablet_bootstrap.cc:654] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.659340 22648 log.cc:824] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:35.660444 22649 tablet_bootstrap.cc:654] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.660744 22650 tablet_bootstrap.cc:492] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:35.661376 22651 tablet_bootstrap.cc:492] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:35.661631 22649 log.cc:824] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:35.662320 22647 tablet_bootstrap.cc:654] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.663020 22650 tablet_bootstrap.cc:654] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.663537 22647 log.cc:824] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:35.663915 22650 log.cc:824] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:35.664443 22651 tablet_bootstrap.cc:654] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.665674 22651 log.cc:824] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:35.670480 22653 tablet_bootstrap.cc:492] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:35.671227 22648 tablet_bootstrap.cc:492] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:35.671892 22648 ts_tablet_manager.cc:1367] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.006s
I20240502 14:14:35.672041 22647 tablet_bootstrap.cc:492] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:35.672039 22649 tablet_bootstrap.cc:492] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:35.672300 22647 ts_tablet_manager.cc:1367] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20240502 14:14:35.672302 22649 ts_tablet_manager.cc:1367] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20240502 14:14:35.674152 22653 tablet_bootstrap.cc:654] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.675405 22653 log.cc:824] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:35.677068 22650 tablet_bootstrap.cc:492] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:35.677346 22650 ts_tablet_manager.cc:1367] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20240502 14:14:35.682096 22649 raft_consensus.cc:348] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:35.682713 22649 raft_consensus.cc:374] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.682885 22649 raft_consensus.cc:729] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.683100 22651 tablet_bootstrap.cc:492] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:35.683359 22651 ts_tablet_manager.cc:1367] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.004s
I20240502 14:14:35.683871 22649 consensus_queue.cc:260] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:35.686636 22648 raft_consensus.cc:348] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:35.687764 22648 raft_consensus.cc:374] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.689282 22653 tablet_bootstrap.cc:492] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:35.689440 22648 raft_consensus.cc:729] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.690387 22648 consensus_queue.cc:260] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:35.691093 22653 ts_tablet_manager.cc:1367] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20240502 14:14:35.691785 22649 ts_tablet_manager.cc:1398] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20240502 14:14:35.692301 22650 raft_consensus.cc:348] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.692662 22650 raft_consensus.cc:374] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.692755 22650 raft_consensus.cc:729] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.693127 22649 tablet_bootstrap.cc:492] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:35.693046 22651 raft_consensus.cc:348] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.693488 22651 raft_consensus.cc:374] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.693372 22650 consensus_queue.cc:260] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.693647 22651 raft_consensus.cc:729] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.694542 22651 consensus_queue.cc:260] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.694795 22649 tablet_bootstrap.cc:654] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.699740 22647 raft_consensus.cc:348] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.700094 22647 raft_consensus.cc:374] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.700184 22647 raft_consensus.cc:729] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.700678 22648 ts_tablet_manager.cc:1398] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.029s user 0.011s sys 0.002s
I20240502 14:14:35.700843 22651 ts_tablet_manager.cc:1398] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20240502 14:14:35.701815 22650 ts_tablet_manager.cc:1398] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.024s user 0.010s sys 0.000s
I20240502 14:14:35.701529 22647 consensus_queue.cc:260] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.703132 22651 tablet_bootstrap.cc:492] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:35.705363 22649 tablet_bootstrap.cc:492] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:35.705529 22649 ts_tablet_manager.cc:1367] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 14:14:35.706530 22649 raft_consensus.cc:348] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.706707 22651 tablet_bootstrap.cc:654] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.706830 22649 raft_consensus.cc:374] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.706969 22649 raft_consensus.cc:729] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.707407 22649 consensus_queue.cc:260] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.708787 22649 ts_tablet_manager.cc:1398] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:35.710685 22651 tablet_bootstrap.cc:492] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:35.710861 22647 ts_tablet_manager.cc:1398] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.038s user 0.012s sys 0.007s
I20240502 14:14:35.710865 22651 ts_tablet_manager.cc:1367] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240502 14:14:35.711331 22647 tablet_bootstrap.cc:492] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:35.709561 22653 raft_consensus.cc:348] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.711848 22651 raft_consensus.cc:348] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.711977 22653 raft_consensus.cc:374] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.712076 22651 raft_consensus.cc:374] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.712193 22651 raft_consensus.cc:729] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.712170 22653 raft_consensus.cc:729] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.712509 22651 consensus_queue.cc:260] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.713055 22651 ts_tablet_manager.cc:1398] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:35.713143 22647 tablet_bootstrap.cc:654] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.713572 22653 consensus_queue.cc:260] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.715409 22653 ts_tablet_manager.cc:1398] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.024s user 0.007s sys 0.005s
I20240502 14:14:35.716318 22647 tablet_bootstrap.cc:492] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:35.716526 22647 ts_tablet_manager.cc:1367] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:35.717481 22647 raft_consensus.cc:348] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:35.717671 22647 raft_consensus.cc:374] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.717767 22647 raft_consensus.cc:729] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.717972 22647 consensus_queue.cc:260] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:35.718369 22647 ts_tablet_manager.cc:1398] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 14:14:35.720465 22230 tablet.cc:2369] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:14:35.730093 22623 tablet.cc:2369] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:35.742352 22661 raft_consensus.cc:482] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:35.742553 22661 raft_consensus.cc:504] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.744024 22661 leader_election.cc:288] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:35.751927 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38de82b0fbb54b8280debbea89a57432" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:35.751958 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38de82b0fbb54b8280debbea89a57432" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:35.752334 22315 raft_consensus.cc:2454] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:35.752338 22446 raft_consensus.cc:2454] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:35.752852 21987 leader_election.cc:302] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fe26b209fc9842d28d31c6243c869061, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:35.753388 22661 raft_consensus.cc:2781] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:35.753501 22661 raft_consensus.cc:482] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:35.753600 22661 raft_consensus.cc:3037] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.755537 22661 raft_consensus.cc:504] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.756068 22661 leader_election.cc:288] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:35.756414 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38de82b0fbb54b8280debbea89a57432" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:35.756467 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38de82b0fbb54b8280debbea89a57432" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:35.756553 22446 raft_consensus.cc:3037] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.756645 22315 raft_consensus.cc:3037] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.758467 22446 raft_consensus.cc:2454] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:35.758592 22315 raft_consensus.cc:2454] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:35.758812 21987 leader_election.cc:302] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fe26b209fc9842d28d31c6243c869061, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:35.759047 22661 raft_consensus.cc:2781] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:35.759469 22661 raft_consensus.cc:686] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [term 1 LEADER]: Becoming Leader. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Running, Role: LEADER
I20240502 14:14:35.759977 22661 consensus_queue.cc:237] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:35.764022 21779 catalog_manager.cc:5484] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26b209fc9842d28d31c6243c869061 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "fe26b209fc9842d28d31c6243c869061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:35.790062 22099 tablet.cc:2369] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:14:35.805176 22492 tablet.cc:2369] T 38de82b0fbb54b8280debbea89a57432 P fe26ecc72bed46179803517c549c5673: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:14:35.860973 21967 tablet.cc:2369] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:35.870546 22664 raft_consensus.cc:482] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:35.870803 22664 raft_consensus.cc:504] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.872457 22664 leader_election.cc:288] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:35.878837 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c3ece6f2614827b2b5abd95228ac6d" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:35.879046 22446 raft_consensus.cc:2454] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:35.879166 21921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c3ece6f2614827b2b5abd95228ac6d" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:35.879619 22118 leader_election.cc:302] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:35.879675 21921 raft_consensus.cc:2454] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:35.879971 22664 raft_consensus.cc:2781] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:35.880111 22664 raft_consensus.cc:482] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:35.880219 22664 raft_consensus.cc:3037] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.882266 22664 raft_consensus.cc:504] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.882882 22664 leader_election.cc:288] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:35.883164 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c3ece6f2614827b2b5abd95228ac6d" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:35.883402 22446 raft_consensus.cc:3037] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.885084 22446 raft_consensus.cc:2454] T b5c3ece6f2614827b2b5abd95228ac6d P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:35.885350 21921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c3ece6f2614827b2b5abd95228ac6d" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:35.885406 22118 leader_election.cc:302] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:35.885500 21921 raft_consensus.cc:3037] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.885634 22664 raft_consensus.cc:2781] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:35.886022 22664 raft_consensus.cc:686] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Running, Role: LEADER
I20240502 14:14:35.886505 22664 consensus_queue.cc:237] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:35.888190 21921 raft_consensus.cc:2454] T b5c3ece6f2614827b2b5abd95228ac6d P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
W20240502 14:14:35.889001 22361 tablet.cc:2369] T 38de82b0fbb54b8280debbea89a57432 P 597f2b4f10754298a4a428ccd386c7d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:35.891866 21779 catalog_manager.cc:5484] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:36.027230 22660 raft_consensus.cc:482] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.027483 22660 raft_consensus.cc:504] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:36.029098 22660 leader_election.cc:288] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:36.036821 21921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e999c7d734334277b470ac0ebbee304f" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:36.037070 21921 raft_consensus.cc:2454] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:36.037628 22510 leader_election.cc:302] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 4b0a27c175a3442dac0cae3f5b505110; no voters:
I20240502 14:14:36.038004 22660 raft_consensus.cc:2781] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.038127 22660 raft_consensus.cc:482] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.038219 22660 raft_consensus.cc:3037] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.038491 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e999c7d734334277b470ac0ebbee304f" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:36.038918 22184 raft_consensus.cc:2454] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:36.040657 22660 raft_consensus.cc:504] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:36.041319 22660 leader_election.cc:288] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:36.041708 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e999c7d734334277b470ac0ebbee304f" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:36.041891 22184 raft_consensus.cc:3037] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.042057 21921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e999c7d734334277b470ac0ebbee304f" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:36.042245 21921 raft_consensus.cc:3037] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.044124 21921 raft_consensus.cc:2454] T e999c7d734334277b470ac0ebbee304f P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:36.044463 22510 leader_election.cc:302] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 4b0a27c175a3442dac0cae3f5b505110; no voters:
I20240502 14:14:36.044752 22660 raft_consensus.cc:2781] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:36.044868 22184 raft_consensus.cc:2454] T e999c7d734334277b470ac0ebbee304f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:36.045200 22660 raft_consensus.cc:686] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:36.045702 22660 consensus_queue.cc:237] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:36.050184 21779 catalog_manager.cc:5484] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:36.171393 22661 consensus_queue.cc:1035] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.179457 22661 consensus_queue.cc:1035] T 38de82b0fbb54b8280debbea89a57432 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.302024 22664 consensus_queue.cc:1035] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.324409 22674 consensus_queue.cc:1035] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:36.485323 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:36.485586 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:36.492807 22680 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:14:36.617600 22715 consensus_queue.cc:1035] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.631620 22714 consensus_queue.cc:1035] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 290.676 ms
time per row: 0.145338 ms
I20240502 14:14:36.929998 21777 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:36.939672 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 5492c913c51d43afbf165205127a172e (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.939719 22557 tablet_service.cc:1467] Processing CreateTablet for tablet 5492c913c51d43afbf165205127a172e (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.940147 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5492c913c51d43afbf165205127a172e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.940268 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5492c913c51d43afbf165205127a172e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.944551 22753 tablet_bootstrap.cc:492] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:36.945168 22163 tablet_service.cc:1467] Processing CreateTablet for tablet 84d7d6aac46946bd8c27cc663a0bfa0b (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.945603 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d7d6aac46946bd8c27cc663a0bfa0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.946599 22753 tablet_bootstrap.cc:654] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.947126 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 84d7d6aac46946bd8c27cc663a0bfa0b (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.947700 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d7d6aac46946bd8c27cc663a0bfa0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.948459 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 20e53ba0076940e0942889748251188a (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.948848 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e53ba0076940e0942889748251188a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.953174 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 5492c913c51d43afbf165205127a172e (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.953692 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5492c913c51d43afbf165205127a172e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.955628 22032 tablet_service.cc:1467] Processing CreateTablet for tablet 20e53ba0076940e0942889748251188a (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.956104 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e53ba0076940e0942889748251188a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.959848 22425 tablet_service.cc:1467] Processing CreateTablet for tablet 84d7d6aac46946bd8c27cc663a0bfa0b (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.960103 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 20e53ba0076940e0942889748251188a (DEFAULT_TABLE table=rebalance_test_table_1 [id=0e4d590249b54f2694119799c81d9c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.960335 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d7d6aac46946bd8c27cc663a0bfa0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.960578 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e53ba0076940e0942889748251188a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.963948 22753 tablet_bootstrap.cc:492] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:36.964073 22753 ts_tablet_manager.cc:1367] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.003s
I20240502 14:14:36.965015 22753 raft_consensus.cc:348] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:36.965191 22753 raft_consensus.cc:374] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.965260 22753 raft_consensus.cc:729] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.965503 22753 consensus_queue.cc:260] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:36.971984 22757 tablet_bootstrap.cc:492] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:36.974515 22757 tablet_bootstrap.cc:654] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.002092 22758 tablet_bootstrap.cc:492] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:37.004281 22758 tablet_bootstrap.cc:654] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.005906 22760 tablet_bootstrap.cc:492] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:37.008073 22760 tablet_bootstrap.cc:654] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.010557 22761 tablet_bootstrap.cc:492] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:37.012849 22761 tablet_bootstrap.cc:654] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.015996 22755 tablet_bootstrap.cc:492] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:37.018406 22755 tablet_bootstrap.cc:654] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.021919 22753 ts_tablet_manager.cc:1398] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.058s user 0.009s sys 0.023s
I20240502 14:14:37.022322 22753 tablet_bootstrap.cc:492] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:37.024382 22753 tablet_bootstrap.cc:654] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.027405 22760 tablet_bootstrap.cc:492] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:37.027643 22760 ts_tablet_manager.cc:1367] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20240502 14:14:37.028628 22760 raft_consensus.cc:348] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.028949 22760 raft_consensus.cc:374] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.029095 22760 raft_consensus.cc:729] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.029433 22760 consensus_queue.cc:260] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.033730 22760 ts_tablet_manager.cc:1398] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:37.034739 22758 tablet_bootstrap.cc:492] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:37.034927 22758 ts_tablet_manager.cc:1367] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 14:14:37.035915 22758 raft_consensus.cc:348] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.036187 22758 raft_consensus.cc:374] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.036304 22758 raft_consensus.cc:729] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.036569 22758 consensus_queue.cc:260] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.037182 22758 ts_tablet_manager.cc:1398] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:37.037503 22758 tablet_bootstrap.cc:492] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:37.039045 22756 raft_consensus.cc:482] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:37.039266 22756 raft_consensus.cc:504] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.050036 22761 tablet_bootstrap.cc:492] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:37.050217 22761 ts_tablet_manager.cc:1367] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 14:14:37.051111 22761 raft_consensus.cc:348] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.051359 22761 raft_consensus.cc:374] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.051468 22761 raft_consensus.cc:729] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.051702 22761 consensus_queue.cc:260] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.053344 22756 leader_election.cc:288] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:37.054582 22761 ts_tablet_manager.cc:1398] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:37.058238 22761 tablet_bootstrap.cc:492] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:37.060570 22757 tablet_bootstrap.cc:492] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:37.060802 22757 ts_tablet_manager.cc:1367] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.089s user 0.031s sys 0.054s
I20240502 14:14:37.061753 22757 raft_consensus.cc:348] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.062062 22757 raft_consensus.cc:374] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.062233 22757 raft_consensus.cc:729] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.062546 22757 consensus_queue.cc:260] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.079957 22577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5492c913c51d43afbf165205127a172e" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:37.080382 22577 raft_consensus.cc:2454] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 0.
I20240502 14:14:37.080459 22757 ts_tablet_manager.cc:1398] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.020s user 0.001s sys 0.002s
I20240502 14:14:37.080652 22758 tablet_bootstrap.cc:654] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.081108 22761 tablet_bootstrap.cc:654] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.081126 22249 leader_election.cc:302] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:37.081533 22756 raft_consensus.cc:2781] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:37.081637 22756 raft_consensus.cc:482] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:37.081723 22756 raft_consensus.cc:3037] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.081779 22686 raft_consensus.cc:482] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:37.082067 22686 raft_consensus.cc:504] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.084257 22686 leader_election.cc:288] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:37.084543 22756 raft_consensus.cc:504] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.085178 22756 leader_election.cc:288] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 election: Requested vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:37.086812 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5492c913c51d43afbf165205127a172e" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:37.087330 22053 raft_consensus.cc:2454] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 0.
I20240502 14:14:37.087530 22735 raft_consensus.cc:482] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:37.087707 22735 raft_consensus.cc:504] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.086838 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5492c913c51d43afbf165205127a172e" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:37.088148 22052 raft_consensus.cc:3037] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.088850 22735 leader_election.cc:288] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:37.091550 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5492c913c51d43afbf165205127a172e" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:37.091913 22315 raft_consensus.cc:2379] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe26b209fc9842d28d31c6243c869061 in current term 1: Already voted for candidate 597f2b4f10754298a4a428ccd386c7d3 in this term.
I20240502 14:14:37.094281 22052 raft_consensus.cc:2454] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 1.
I20240502 14:14:37.094739 22249 leader_election.cc:302] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 597f2b4f10754298a4a428ccd386c7d3, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:37.095109 22756 raft_consensus.cc:2781] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:37.085958 22577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5492c913c51d43afbf165205127a172e" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:37.097959 22577 raft_consensus.cc:3037] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.100901 22577 raft_consensus.cc:2454] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 1.
I20240502 14:14:37.101964 22755 tablet_bootstrap.cc:492] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:37.102294 22755 ts_tablet_manager.cc:1367] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20240502 14:14:37.103339 22755 raft_consensus.cc:348] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.103665 22755 raft_consensus.cc:374] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.103807 22755 raft_consensus.cc:729] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.104182 22755 consensus_queue.cc:260] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.104951 22755 ts_tablet_manager.cc:1398] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:37.109910 22758 tablet_bootstrap.cc:492] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:37.111299 22758 ts_tablet_manager.cc:1367] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20240502 14:14:37.111066 22753 tablet_bootstrap.cc:492] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:37.111460 22753 ts_tablet_manager.cc:1367] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.030s
I20240502 14:14:37.112289 22753 raft_consensus.cc:348] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.113272 22753 raft_consensus.cc:374] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.113482 22753 raft_consensus.cc:729] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.113893 22753 consensus_queue.cc:260] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.115399 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7d6aac46946bd8c27cc663a0bfa0b" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:37.115648 22445 raft_consensus.cc:2454] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:37.116235 21855 leader_election.cc:302] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:37.116742 22686 raft_consensus.cc:2781] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:37.116890 22686 raft_consensus.cc:482] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:37.116998 22686 raft_consensus.cc:3037] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.120357 22686 raft_consensus.cc:504] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.121344 22753 ts_tablet_manager.cc:1398] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20240502 14:14:37.121450 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7d6aac46946bd8c27cc663a0bfa0b" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:37.121645 22445 raft_consensus.cc:3037] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.112022 22758 raft_consensus.cc:348] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.123226 22758 raft_consensus.cc:374] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.123342 22758 raft_consensus.cc:729] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.123641 22758 consensus_queue.cc:260] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.124179 22758 ts_tablet_manager.cc:1398] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240502 14:14:37.126587 22445 raft_consensus.cc:2454] T 84d7d6aac46946bd8c27cc663a0bfa0b P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:37.127218 22686 leader_election.cc:288] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:37.127400 21855 leader_election.cc:302] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:37.128779 22756 raft_consensus.cc:686] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [term 1 LEADER]: Becoming Leader. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Running, Role: LEADER
I20240502 14:14:37.129374 22756 consensus_queue.cc:237] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:37.131425 22761 tablet_bootstrap.cc:492] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:37.131575 22761 ts_tablet_manager.cc:1367] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.003s
I20240502 14:14:37.132576 22761 raft_consensus.cc:348] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.132854 22761 raft_consensus.cc:374] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.132956 22761 raft_consensus.cc:729] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.133240 22761 consensus_queue.cc:260] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.133805 22761 ts_tablet_manager.cc:1398] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:37.134119 22686 raft_consensus.cc:2781] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:37.142941 22577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5492c913c51d43afbf165205127a172e" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:37.143317 22577 raft_consensus.cc:2379] T 5492c913c51d43afbf165205127a172e P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe26b209fc9842d28d31c6243c869061 in current term 1: Already voted for candidate 597f2b4f10754298a4a428ccd386c7d3 in this term.
I20240502 14:14:37.144052 21987 leader_election.cc:302] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe26b209fc9842d28d31c6243c869061; no voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3
I20240502 14:14:37.144587 22735 raft_consensus.cc:2726] T 5492c913c51d43afbf165205127a172e P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:37.145769 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7d6aac46946bd8c27cc663a0bfa0b" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:37.145879 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7d6aac46946bd8c27cc663a0bfa0b" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:37.146143 22184 raft_consensus.cc:2454] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:37.147827 22686 raft_consensus.cc:686] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Running, Role: LEADER
I20240502 14:14:37.148363 22686 consensus_queue.cc:237] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:37.150525 21777 catalog_manager.cc:5484] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "597f2b4f10754298a4a428ccd386c7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:37.151748 21780 catalog_manager.cc:5484] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "4b0a27c175a3442dac0cae3f5b505110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:37.324725 22724 raft_consensus.cc:482] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:37.324877 22724 raft_consensus.cc:504] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.325531 22724 leader_election.cc:288] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:37.331233 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e53ba0076940e0942889748251188a" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:37.331403 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e53ba0076940e0942889748251188a" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:37.331488 22315 raft_consensus.cc:2454] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:37.331612 22052 raft_consensus.cc:2454] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:37.331882 22513 leader_election.cc:302] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:37.332149 22724 raft_consensus.cc:2781] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:37.332243 22724 raft_consensus.cc:482] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:37.332304 22724 raft_consensus.cc:3037] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.334159 22724 raft_consensus.cc:504] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.334625 22724 leader_election.cc:288] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:37.335042 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e53ba0076940e0942889748251188a" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:37.335027 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e53ba0076940e0942889748251188a" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:37.335269 22052 raft_consensus.cc:3037] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.335269 22315 raft_consensus.cc:3037] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.337942 22052 raft_consensus.cc:2454] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:37.337942 22315 raft_consensus.cc:2454] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:37.338393 22513 leader_election.cc:302] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:37.338640 22724 raft_consensus.cc:2781] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:37.338930 22724 raft_consensus.cc:686] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:37.339401 22724 consensus_queue.cc:237] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:37.342182 21777 catalog_manager.cc:5484] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:37.536077 22756 consensus_queue.cc:1035] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.537523 22756 consensus_queue.cc:1035] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.555006 22686 consensus_queue.cc:1035] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.560271 22183 raft_consensus.cc:3037] T 84d7d6aac46946bd8c27cc663a0bfa0b P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.565654 22779 consensus_queue.cc:1035] T 84d7d6aac46946bd8c27cc663a0bfa0b P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:37.764951 22787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:37.765224 22787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:37.772403 22787 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:14:37.922806 22052 raft_consensus.cc:1261] T 20e53ba0076940e0942889748251188a P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Refusing update from remote peer 491bfc422e6440749f54ffffdb717e3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:37.923633 22724 consensus_queue.cc:1035] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.953856 22315 raft_consensus.cc:1261] T 20e53ba0076940e0942889748251188a P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Refusing update from remote peer 491bfc422e6440749f54ffffdb717e3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:37.962726 22831 consensus_queue.cc:1035] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.968768 22791 mvcc.cc:205] Tried to move back new op lower bound from 7023244402355830784 to 7023244399984164864. Current Snapshot: MvccSnapshot[applied={T|T < 7023244402355830784}]
I20240502 14:14:38.011471 22833 mvcc.cc:205] Tried to move back new op lower bound from 7023244402355830784 to 7023244399984164864. Current Snapshot: MvccSnapshot[applied={T|T < 7023244402355830784 or (T in {7023244402355830784})}]
INSERT report
rows total: 2000
time total: 291.002 ms
time per row: 0.145501 ms
I20240502 14:14:38.206969 21775 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:38.215613 22425 tablet_service.cc:1467] Processing CreateTablet for tablet a33dd28a2c974ce4b16ae21107ccdc7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.215631 22163 tablet_service.cc:1467] Processing CreateTablet for tablet a33dd28a2c974ce4b16ae21107ccdc7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.215842 21901 tablet_service.cc:1467] Processing CreateTablet for tablet a33dd28a2c974ce4b16ae21107ccdc7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.216053 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33dd28a2c974ce4b16ae21107ccdc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.216053 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33dd28a2c974ce4b16ae21107ccdc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.216256 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33dd28a2c974ce4b16ae21107ccdc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.220372 22557 tablet_service.cc:1467] Processing CreateTablet for tablet a188e47225bc4fafa214540472d88d26 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.220824 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a188e47225bc4fafa214540472d88d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.220877 22033 tablet_service.cc:1467] Processing CreateTablet for tablet a188e47225bc4fafa214540472d88d26 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.221285 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a188e47225bc4fafa214540472d88d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.224184 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 1df7a948dbd24e29812175b51c325ca2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.224658 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df7a948dbd24e29812175b51c325ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.227770 21900 tablet_service.cc:1467] Processing CreateTablet for tablet 1df7a948dbd24e29812175b51c325ca2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.228034 22294 tablet_service.cc:1467] Processing CreateTablet for tablet a188e47225bc4fafa214540472d88d26 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.228209 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df7a948dbd24e29812175b51c325ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.228529 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a188e47225bc4fafa214540472d88d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.232758 22164 tablet_service.cc:1467] Processing CreateTablet for tablet 1df7a948dbd24e29812175b51c325ca2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68c0f93b246a42958add12c048fe5091]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.232777 22855 tablet_bootstrap.cc:492] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:38.233155 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df7a948dbd24e29812175b51c325ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.234934 22855 tablet_bootstrap.cc:654] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.252619 22859 tablet_bootstrap.cc:492] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:38.253535 22857 tablet_bootstrap.cc:492] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:38.255142 22859 tablet_bootstrap.cc:654] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.255483 22857 tablet_bootstrap.cc:654] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.256984 22854 tablet_bootstrap.cc:492] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:38.259361 22859 tablet_bootstrap.cc:492] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:38.259495 22854 tablet_bootstrap.cc:654] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.259513 22859 ts_tablet_manager.cc:1367] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:38.260337 22859 raft_consensus.cc:348] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.261669 22860 tablet_bootstrap.cc:492] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:38.262665 22859 raft_consensus.cc:374] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.262811 22859 raft_consensus.cc:729] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.262939 22860 tablet_bootstrap.cc:654] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.263092 22859 consensus_queue.cc:260] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.264037 22854 tablet_bootstrap.cc:492] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:38.264040 22857 tablet_bootstrap.cc:492] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:38.264179 22857 ts_tablet_manager.cc:1367] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20240502 14:14:38.264259 22858 tablet_bootstrap.cc:492] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:38.264602 22859 ts_tablet_manager.cc:1398] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:38.264226 22854 ts_tablet_manager.cc:1367] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:14:38.264974 22857 raft_consensus.cc:348] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.265266 22857 raft_consensus.cc:374] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.265398 22857 raft_consensus.cc:729] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.265715 22857 consensus_queue.cc:260] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.266835 22858 tablet_bootstrap.cc:654] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.268800 22854 raft_consensus.cc:348] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.269060 22854 raft_consensus.cc:374] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.269196 22854 raft_consensus.cc:729] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.269484 22854 consensus_queue.cc:260] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.270097 22860 tablet_bootstrap.cc:492] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:38.270095 22854 ts_tablet_manager.cc:1398] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:38.270283 22860 ts_tablet_manager.cc:1367] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:14:38.270465 22854 tablet_bootstrap.cc:492] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:38.271123 22860 raft_consensus.cc:348] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.271330 22860 raft_consensus.cc:374] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.271423 22860 raft_consensus.cc:729] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.272001 22857 ts_tablet_manager.cc:1398] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s
I20240502 14:14:38.272245 22857 tablet_bootstrap.cc:492] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:38.273139 22854 tablet_bootstrap.cc:654] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.273774 22857 tablet_bootstrap.cc:654] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.277590 22857 tablet_bootstrap.cc:492] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:38.276219 22860 consensus_queue.cc:260] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.277794 22857 ts_tablet_manager.cc:1367] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 14:14:38.278311 22860 ts_tablet_manager.cc:1398] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 14:14:38.279357 22857 raft_consensus.cc:348] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.279568 22857 raft_consensus.cc:374] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.279659 22857 raft_consensus.cc:729] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.279668 22855 tablet_bootstrap.cc:492] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:38.279814 22855 ts_tablet_manager.cc:1367] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.047s user 0.017s sys 0.017s
I20240502 14:14:38.279915 22857 consensus_queue.cc:260] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.280424 22857 ts_tablet_manager.cc:1398] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:38.281131 22855 raft_consensus.cc:348] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.281343 22855 raft_consensus.cc:374] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.281348 22858 tablet_bootstrap.cc:492] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:38.281450 22855 raft_consensus.cc:729] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.281488 22858 ts_tablet_manager.cc:1367] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240502 14:14:38.281728 22855 consensus_queue.cc:260] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.282374 22855 ts_tablet_manager.cc:1398] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:38.282369 22858 raft_consensus.cc:348] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.282683 22855 tablet_bootstrap.cc:492] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:38.284232 22855 tablet_bootstrap.cc:654] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.284371 22846 raft_consensus.cc:482] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.284538 22846 raft_consensus.cc:504] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.285439 22846 leader_election.cc:288] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:38.285487 22858 raft_consensus.cc:374] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.285593 22858 raft_consensus.cc:729] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.285863 22858 consensus_queue.cc:260] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.286221 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a188e47225bc4fafa214540472d88d26" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:38.286399 22855 tablet_bootstrap.cc:492] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:38.286473 22858 ts_tablet_manager.cc:1398] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:38.286554 22855 ts_tablet_manager.cc:1367] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:38.287245 22855 raft_consensus.cc:348] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.287431 22855 raft_consensus.cc:374] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.287500 22855 raft_consensus.cc:729] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.287564 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a188e47225bc4fafa214540472d88d26" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:38.287763 22053 raft_consensus.cc:2454] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:38.287742 22855 consensus_queue.cc:260] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.288225 22855 ts_tablet_manager.cc:1398] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:38.288404 22511 leader_election.cc:302] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:38.288647 22846 raft_consensus.cc:2781] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.288756 22846 raft_consensus.cc:482] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.288864 22846 raft_consensus.cc:3037] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.288897 22854 tablet_bootstrap.cc:492] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:38.289033 22854 ts_tablet_manager.cc:1367] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.011s
I20240502 14:14:38.289192 22315 raft_consensus.cc:2454] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:38.289772 22854 raft_consensus.cc:348] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.290047 22854 raft_consensus.cc:374] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.290122 22854 raft_consensus.cc:729] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.290922 22854 consensus_queue.cc:260] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.291368 22846 raft_consensus.cc:504] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.292268 22854 ts_tablet_manager.cc:1398] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:38.292393 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a188e47225bc4fafa214540472d88d26" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:38.292569 22053 raft_consensus.cc:3037] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.292588 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a188e47225bc4fafa214540472d88d26" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:38.292769 22315 raft_consensus.cc:3037] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.294576 22053 raft_consensus.cc:2454] T a188e47225bc4fafa214540472d88d26 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:38.294656 22315 raft_consensus.cc:2454] T a188e47225bc4fafa214540472d88d26 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:38.294852 22846 leader_election.cc:288] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:38.294974 22511 leader_election.cc:302] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:38.295308 22846 raft_consensus.cc:2781] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.295496 22846 raft_consensus.cc:686] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:38.295791 22846 consensus_queue.cc:237] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:38.298252 21780 catalog_manager.cc:5484] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:38.344095 22690 raft_consensus.cc:482] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.344234 22690 raft_consensus.cc:504] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.344827 22690 leader_election.cc:288] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:38.345273 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33dd28a2c974ce4b16ae21107ccdc7f" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:38.345361 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33dd28a2c974ce4b16ae21107ccdc7f" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:38.345474 22445 raft_consensus.cc:2454] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:38.345580 21919 raft_consensus.cc:2454] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:38.345938 22118 leader_election.cc:302] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:38.346202 22690 raft_consensus.cc:2781] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.346312 22690 raft_consensus.cc:482] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.346405 22690 raft_consensus.cc:3037] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.348224 22690 raft_consensus.cc:504] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.348721 22690 leader_election.cc:288] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:38.349050 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33dd28a2c974ce4b16ae21107ccdc7f" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:38.349141 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33dd28a2c974ce4b16ae21107ccdc7f" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:38.349279 22445 raft_consensus.cc:3037] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.349299 21919 raft_consensus.cc:3037] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.351996 22445 raft_consensus.cc:2454] T a33dd28a2c974ce4b16ae21107ccdc7f P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:38.352023 21919 raft_consensus.cc:2454] T a33dd28a2c974ce4b16ae21107ccdc7f P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:38.352378 22118 leader_election.cc:302] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:38.352717 22690 raft_consensus.cc:2781] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.352934 22690 raft_consensus.cc:686] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Running, Role: LEADER
I20240502 14:14:38.353283 22690 consensus_queue.cc:237] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:38.355728 21777 catalog_manager.cc:5484] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:38.433151 22865 raft_consensus.cc:482] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.433348 22865 raft_consensus.cc:504] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.434723 22865 leader_election.cc:288] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:38.442386 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df7a948dbd24e29812175b51c325ca2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:38.442621 21919 raft_consensus.cc:2454] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:38.442624 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df7a948dbd24e29812175b51c325ca2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:38.442842 22184 raft_consensus.cc:2454] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:38.443054 22379 leader_election.cc:302] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:38.443432 22865 raft_consensus.cc:2781] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.443545 22865 raft_consensus.cc:482] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.443655 22865 raft_consensus.cc:3037] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.445487 22865 raft_consensus.cc:504] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.445955 22865 leader_election.cc:288] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:38.446338 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df7a948dbd24e29812175b51c325ca2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:38.446352 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df7a948dbd24e29812175b51c325ca2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:38.446502 22184 raft_consensus.cc:3037] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.446525 21919 raft_consensus.cc:3037] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.448244 22184 raft_consensus.cc:2454] T 1df7a948dbd24e29812175b51c325ca2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:38.448328 21919 raft_consensus.cc:2454] T 1df7a948dbd24e29812175b51c325ca2 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:38.448599 22381 leader_election.cc:302] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:38.448881 22865 raft_consensus.cc:2781] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.449246 22865 raft_consensus.cc:686] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [term 1 LEADER]: Becoming Leader. State: Replica: fe26ecc72bed46179803517c549c5673, State: Running, Role: LEADER
I20240502 14:14:38.449631 22865 consensus_queue.cc:237] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:38.452637 21777 catalog_manager.cc:5484] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26ecc72bed46179803517c549c5673 (127.21.58.69). New cstate: current_term: 1 leader_uuid: "fe26ecc72bed46179803517c549c5673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:38.777669 22690 consensus_queue.cc:1035] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.784269 22886 consensus_queue.cc:1035] T a33dd28a2c974ce4b16ae21107ccdc7f P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:38.785879 22879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:38.786425 22879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:38.795027 22879 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:14:38.860760 22865 consensus_queue.cc:1035] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.863905 22846 consensus_queue.cc:1035] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.864352 22865 consensus_queue.cc:1035] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.924069 22909 consensus_queue.cc:1035] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 285.163 ms
time per row: 0.142581 ms
I20240502 14:14:39.223244 21780 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:39.231388 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 3f92441b03ba422e9228060703d78625 (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.231864 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f92441b03ba422e9228060703d78625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.232283 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 40ec36463b2e43b887a2da7747de152e (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.232735 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ec36463b2e43b887a2da7747de152e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.232663 21901 tablet_service.cc:1467] Processing CreateTablet for tablet dce65a3c4f1744a38cadc96f79c0c084 (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.233071 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce65a3c4f1744a38cadc96f79c0c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.233711 22032 tablet_service.cc:1467] Processing CreateTablet for tablet 40ec36463b2e43b887a2da7747de152e (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.234108 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ec36463b2e43b887a2da7747de152e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.238958 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 40ec36463b2e43b887a2da7747de152e (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.239483 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ec36463b2e43b887a2da7747de152e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.243640 22557 tablet_service.cc:1467] Processing CreateTablet for tablet 3f92441b03ba422e9228060703d78625 (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.244102 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f92441b03ba422e9228060703d78625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.246415 22940 tablet_bootstrap.cc:492] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:39.247596 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 3f92441b03ba422e9228060703d78625 (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.248020 22942 tablet_bootstrap.cc:492] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:39.248106 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f92441b03ba422e9228060703d78625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.248699 22940 tablet_bootstrap.cc:654] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.249785 22942 tablet_bootstrap.cc:654] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.253527 22426 tablet_service.cc:1467] Processing CreateTablet for tablet dce65a3c4f1744a38cadc96f79c0c084 (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.254060 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce65a3c4f1744a38cadc96f79c0c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.253937 22164 tablet_service.cc:1467] Processing CreateTablet for tablet dce65a3c4f1744a38cadc96f79c0c084 (DEFAULT_TABLE table=rebalance_test_table_3 [id=49ce654f56f248e58e24df64094142e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.254454 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce65a3c4f1744a38cadc96f79c0c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.263170 22940 tablet_bootstrap.cc:492] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:39.263424 22940 ts_tablet_manager.cc:1367] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240502 14:14:39.264503 22940 raft_consensus.cc:348] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.264899 22940 raft_consensus.cc:374] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.265064 22940 raft_consensus.cc:729] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.265483 22940 consensus_queue.cc:260] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.266230 22940 ts_tablet_manager.cc:1398] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:39.282296 22940 tablet_bootstrap.cc:492] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:39.284384 22940 tablet_bootstrap.cc:654] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.286945 22948 tablet_bootstrap.cc:492] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:39.289228 22948 tablet_bootstrap.cc:654] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.289803 22946 tablet_bootstrap.cc:492] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:39.293810 22946 tablet_bootstrap.cc:654] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.295300 22941 tablet_bootstrap.cc:492] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:39.295897 22945 tablet_bootstrap.cc:492] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:39.296317 22940 tablet_bootstrap.cc:492] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:39.296523 22940 ts_tablet_manager.cc:1367] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240502 14:14:39.297436 22942 tablet_bootstrap.cc:492] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:39.297336 22940 raft_consensus.cc:348] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.297536 22940 raft_consensus.cc:374] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.297585 22942 ts_tablet_manager.cc:1367] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.050s user 0.027s sys 0.014s
I20240502 14:14:39.297602 22940 raft_consensus.cc:729] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.297876 22940 consensus_queue.cc:260] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.298017 22941 tablet_bootstrap.cc:654] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.298403 22942 raft_consensus.cc:348] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.298609 22942 raft_consensus.cc:374] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.298713 22942 raft_consensus.cc:729] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.299158 22940 ts_tablet_manager.cc:1398] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:39.298990 22942 consensus_queue.cc:260] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.300215 22942 ts_tablet_manager.cc:1398] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:39.301523 22941 tablet_bootstrap.cc:492] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:39.301529 22946 tablet_bootstrap.cc:492] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:39.301651 22941 ts_tablet_manager.cc:1367] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:14:39.301673 22946 ts_tablet_manager.cc:1367] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240502 14:14:39.302505 22948 tablet_bootstrap.cc:492] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:39.302454 22946 raft_consensus.cc:348] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.302644 22948 ts_tablet_manager.cc:1367] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20240502 14:14:39.302664 22946 raft_consensus.cc:374] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.302762 22946 raft_consensus.cc:729] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.303017 22946 consensus_queue.cc:260] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.303428 22948 raft_consensus.cc:348] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.303637 22948 raft_consensus.cc:374] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.303736 22946 ts_tablet_manager.cc:1398] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:39.303755 22948 raft_consensus.cc:729] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.304023 22946 tablet_bootstrap.cc:492] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:39.304534 22948 consensus_queue.cc:260] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.305107 22948 ts_tablet_manager.cc:1398] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:39.305332 22946 tablet_bootstrap.cc:654] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.306005 22941 raft_consensus.cc:348] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.306236 22941 raft_consensus.cc:374] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.306346 22941 raft_consensus.cc:729] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.306598 22941 consensus_queue.cc:260] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.307801 22941 ts_tablet_manager.cc:1398] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:39.308084 22941 tablet_bootstrap.cc:492] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:39.309574 22941 tablet_bootstrap.cc:654] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.314077 22941 tablet_bootstrap.cc:492] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:39.314273 22941 ts_tablet_manager.cc:1367] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 14:14:39.314599 22945 tablet_bootstrap.cc:654] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.315184 22941 raft_consensus.cc:348] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.315445 22941 raft_consensus.cc:374] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.315568 22941 raft_consensus.cc:729] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.315954 22941 consensus_queue.cc:260] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.316687 22941 ts_tablet_manager.cc:1398] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:39.319862 22945 tablet_bootstrap.cc:492] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:39.319999 22945 ts_tablet_manager.cc:1367] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 14:14:39.320708 22946 tablet_bootstrap.cc:492] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:39.320861 22946 ts_tablet_manager.cc:1367] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240502 14:14:39.320859 22945 raft_consensus.cc:348] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.321075 22945 raft_consensus.cc:374] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.321170 22945 raft_consensus.cc:729] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.321419 22945 consensus_queue.cc:260] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.321604 22946 raft_consensus.cc:348] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.321784 22946 raft_consensus.cc:374] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.321853 22946 raft_consensus.cc:729] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.322006 22945 ts_tablet_manager.cc:1398] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:39.322114 22946 consensus_queue.cc:260] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.323012 22946 ts_tablet_manager.cc:1398] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:39.382717 22735 raft_consensus.cc:482] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.382894 22735 raft_consensus.cc:504] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.383546 22735 leader_election.cc:288] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:39.383965 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ec36463b2e43b887a2da7747de152e" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:39.384150 22576 raft_consensus.cc:2454] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:39.384112 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ec36463b2e43b887a2da7747de152e" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:39.384325 22314 raft_consensus.cc:2454] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:39.384582 21987 leader_election.cc:302] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:39.384852 22735 raft_consensus.cc:2781] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.384970 22735 raft_consensus.cc:482] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.385058 22735 raft_consensus.cc:3037] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.386876 22735 raft_consensus.cc:504] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.387456 22735 leader_election.cc:288] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Requested vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:39.387801 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ec36463b2e43b887a2da7747de152e" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:39.387876 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ec36463b2e43b887a2da7747de152e" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:39.387972 22576 raft_consensus.cc:3037] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.388048 22314 raft_consensus.cc:3037] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.389762 22576 raft_consensus.cc:2454] T 40ec36463b2e43b887a2da7747de152e P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:39.389889 22314 raft_consensus.cc:2454] T 40ec36463b2e43b887a2da7747de152e P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:39.390121 21987 leader_election.cc:302] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:39.390436 22735 raft_consensus.cc:2781] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.390815 22735 raft_consensus.cc:686] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [term 1 LEADER]: Becoming Leader. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Running, Role: LEADER
I20240502 14:14:39.391131 22735 consensus_queue.cc:237] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:39.393445 21780 catalog_manager.cc:5484] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26b209fc9842d28d31c6243c869061 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "fe26b209fc9842d28d31c6243c869061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:39.525115 22865 raft_consensus.cc:482] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.525298 22865 raft_consensus.cc:504] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.525861 22865 leader_election.cc:288] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:39.526301 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce65a3c4f1744a38cadc96f79c0c084" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:39.526350 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce65a3c4f1744a38cadc96f79c0c084" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:39.526510 21920 raft_consensus.cc:2454] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:39.526535 22184 raft_consensus.cc:2454] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:39.526883 22379 leader_election.cc:302] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:39.527149 22865 raft_consensus.cc:2781] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.527230 22865 raft_consensus.cc:482] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.527300 22865 raft_consensus.cc:3037] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.529103 22865 raft_consensus.cc:504] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.529579 22865 leader_election.cc:288] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:39.529935 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce65a3c4f1744a38cadc96f79c0c084" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:39.529995 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce65a3c4f1744a38cadc96f79c0c084" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:39.530114 21920 raft_consensus.cc:3037] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.530148 22184 raft_consensus.cc:3037] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.532747 22184 raft_consensus.cc:2454] T dce65a3c4f1744a38cadc96f79c0c084 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:39.532773 21920 raft_consensus.cc:2454] T dce65a3c4f1744a38cadc96f79c0c084 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:39.533107 22381 leader_election.cc:302] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:39.533329 22865 raft_consensus.cc:2781] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.533468 22865 raft_consensus.cc:686] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [term 1 LEADER]: Becoming Leader. State: Replica: fe26ecc72bed46179803517c549c5673, State: Running, Role: LEADER
I20240502 14:14:39.533742 22865 consensus_queue.cc:237] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:39.536190 21780 catalog_manager.cc:5484] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26ecc72bed46179803517c549c5673 (127.21.58.69). New cstate: current_term: 1 leader_uuid: "fe26ecc72bed46179803517c549c5673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:39.553200 22777 raft_consensus.cc:482] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.553323 22777 raft_consensus.cc:504] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.553879 22777 leader_election.cc:288] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:39.554293 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f92441b03ba422e9228060703d78625" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:39.554332 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f92441b03ba422e9228060703d78625" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:39.554499 22053 raft_consensus.cc:2454] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 0.
I20240502 14:14:39.554527 22576 raft_consensus.cc:2454] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 0.
I20240502 14:14:39.554893 22249 leader_election.cc:302] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 597f2b4f10754298a4a428ccd386c7d3, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:39.555156 22777 raft_consensus.cc:2781] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.555251 22777 raft_consensus.cc:482] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.555333 22777 raft_consensus.cc:3037] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.557057 22777 raft_consensus.cc:504] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.557459 22777 leader_election.cc:288] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 election: Requested vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:39.557828 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f92441b03ba422e9228060703d78625" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:39.557868 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f92441b03ba422e9228060703d78625" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:39.557984 22053 raft_consensus.cc:3037] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.558043 22576 raft_consensus.cc:3037] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.560104 22053 raft_consensus.cc:2454] T 3f92441b03ba422e9228060703d78625 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 1.
I20240502 14:14:39.560127 22576 raft_consensus.cc:2454] T 3f92441b03ba422e9228060703d78625 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 1.
I20240502 14:14:39.560412 22249 leader_election.cc:302] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 597f2b4f10754298a4a428ccd386c7d3, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:39.560659 22777 raft_consensus.cc:2781] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.560987 22777 raft_consensus.cc:686] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 LEADER]: Becoming Leader. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Running, Role: LEADER
I20240502 14:14:39.561261 22777 consensus_queue.cc:237] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:39.563556 21780 catalog_manager.cc:5484] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "597f2b4f10754298a4a428ccd386c7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:39.798211 22735 consensus_queue.cc:1035] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.804828 22735 consensus_queue.cc:1035] T 40ec36463b2e43b887a2da7747de152e P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:39.888710 22960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:39.889014 22960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:39.896083 22960 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:14:39.995067 22865 consensus_queue.cc:1035] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.007300 22777 consensus_queue.cc:1035] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.036909 22865 consensus_queue.cc:1035] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.059408 22958 consensus_queue.cc:1035] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.322 ms
time per row: 0.126661 ms
I20240502 14:14:40.296924 21779 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:40.306632 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 2727235f406a456cbff925bda1f1cddb (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.306643 22556 tablet_service.cc:1467] Processing CreateTablet for tablet a51eb2447a234f44b89daf1af6d1ea87 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.306850 22295 tablet_service.cc:1467] Processing CreateTablet for tablet a51eb2447a234f44b89daf1af6d1ea87 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.307122 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a51eb2447a234f44b89daf1af6d1ea87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.307166 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2727235f406a456cbff925bda1f1cddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.307292 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a51eb2447a234f44b89daf1af6d1ea87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.307307 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 2727235f406a456cbff925bda1f1cddb (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.307739 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2727235f406a456cbff925bda1f1cddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.312037 22032 tablet_service.cc:1467] Processing CreateTablet for tablet a51eb2447a234f44b89daf1af6d1ea87 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.312069 22164 tablet_service.cc:1467] Processing CreateTablet for tablet 2727235f406a456cbff925bda1f1cddb (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.312484 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a51eb2447a234f44b89daf1af6d1ea87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.312518 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2727235f406a456cbff925bda1f1cddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.315462 22163 tablet_service.cc:1467] Processing CreateTablet for tablet 81c891f8b1e54990b803493f459151ca (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.315913 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c891f8b1e54990b803493f459151ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.324155 21900 tablet_service.cc:1467] Processing CreateTablet for tablet 81c891f8b1e54990b803493f459151ca (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.324525 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c891f8b1e54990b803493f459151ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.327711 22425 tablet_service.cc:1467] Processing CreateTablet for tablet 81c891f8b1e54990b803493f459151ca (DEFAULT_TABLE table=rebalance_test_table_4 [id=8204feeb035447738e5c19c5462a1c57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.328193 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c891f8b1e54990b803493f459151ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.333551 23030 tablet_bootstrap.cc:492] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:40.334061 23031 tablet_bootstrap.cc:492] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:40.335841 23030 tablet_bootstrap.cc:654] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.337160 23028 tablet_bootstrap.cc:492] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:40.337742 23031 tablet_bootstrap.cc:654] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.337875 23027 tablet_bootstrap.cc:492] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:40.339870 23027 tablet_bootstrap.cc:654] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.340569 23033 tablet_bootstrap.cc:492] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:40.342553 23033 tablet_bootstrap.cc:654] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.342904 23029 tablet_bootstrap.cc:492] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:40.344553 23031 tablet_bootstrap.cc:492] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:40.344687 23029 tablet_bootstrap.cc:654] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.345968 23031 ts_tablet_manager.cc:1367] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20240502 14:14:40.346222 23030 tablet_bootstrap.cc:492] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:40.346352 23030 ts_tablet_manager.cc:1367] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 14:14:40.346796 23031 raft_consensus.cc:348] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.346987 23031 raft_consensus.cc:374] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.347106 23031 raft_consensus.cc:729] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.346971 23030 raft_consensus.cc:348] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.347188 23030 raft_consensus.cc:374] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.347308 23030 raft_consensus.cc:729] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.347482 23031 consensus_queue.cc:260] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.347780 23029 tablet_bootstrap.cc:492] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:40.347662 23030 consensus_queue.cc:260] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.347918 23029 ts_tablet_manager.cc:1367] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:40.348654 23029 raft_consensus.cc:348] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.349231 23030 ts_tablet_manager.cc:1398] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:40.349486 23030 tablet_bootstrap.cc:492] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:40.350293 23028 tablet_bootstrap.cc:654] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.350950 23030 tablet_bootstrap.cc:654] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.351236 23029 raft_consensus.cc:374] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.351364 23029 raft_consensus.cc:729] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.351591 23029 consensus_queue.cc:260] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.352138 23029 ts_tablet_manager.cc:1398] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:40.358156 23031 ts_tablet_manager.cc:1398] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 14:14:40.359618 23033 tablet_bootstrap.cc:492] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:40.360680 23031 tablet_bootstrap.cc:492] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:40.360772 23027 tablet_bootstrap.cc:492] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:40.360949 23027 ts_tablet_manager.cc:1367] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 14:14:40.361670 23027 raft_consensus.cc:348] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.361855 23027 raft_consensus.cc:374] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.361940 23027 raft_consensus.cc:729] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.362474 23031 tablet_bootstrap.cc:654] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.362581 23033 ts_tablet_manager.cc:1367] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20240502 14:14:40.363198 23028 tablet_bootstrap.cc:492] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:40.363363 23028 ts_tablet_manager.cc:1367] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20240502 14:14:40.363356 23033 raft_consensus.cc:348] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.363591 23033 raft_consensus.cc:374] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.363698 23033 raft_consensus.cc:729] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.364049 23027 consensus_queue.cc:260] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.364457 23033 consensus_queue.cc:260] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.364588 23028 raft_consensus.cc:348] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.365119 23027 ts_tablet_manager.cc:1398] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:40.365608 23033 ts_tablet_manager.cc:1398] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:40.364946 23028 raft_consensus.cc:374] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.365834 23028 raft_consensus.cc:729] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.366370 23028 consensus_queue.cc:260] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.366896 23030 tablet_bootstrap.cc:492] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:40.366885 23028 ts_tablet_manager.cc:1398] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:40.367116 23030 ts_tablet_manager.cc:1367] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20240502 14:14:40.367169 23028 tablet_bootstrap.cc:492] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:40.368288 23030 raft_consensus.cc:348] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.368616 23030 raft_consensus.cc:374] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.368743 23030 raft_consensus.cc:729] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.368969 23030 consensus_queue.cc:260] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.369305 23028 tablet_bootstrap.cc:654] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.369611 23030 ts_tablet_manager.cc:1398] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:40.373241 23031 tablet_bootstrap.cc:492] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:40.373459 23031 ts_tablet_manager.cc:1367] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240502 14:14:40.374439 23031 raft_consensus.cc:348] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.374732 23031 raft_consensus.cc:374] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.374830 23031 raft_consensus.cc:729] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.375150 23031 consensus_queue.cc:260] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.375665 23031 ts_tablet_manager.cc:1398] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 14:14:40.376156 23028 tablet_bootstrap.cc:492] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:40.376288 23028 ts_tablet_manager.cc:1367] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240502 14:14:40.376835 23028 raft_consensus.cc:348] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.377009 23028 raft_consensus.cc:374] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.377081 23028 raft_consensus.cc:729] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.377290 23028 consensus_queue.cc:260] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.377759 23028 ts_tablet_manager.cc:1398] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:40.430112 22690 raft_consensus.cc:482] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:40.430248 22690 raft_consensus.cc:504] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.430744 22690 leader_election.cc:288] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:40.431208 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c891f8b1e54990b803493f459151ca" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:40.431346 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c891f8b1e54990b803493f459151ca" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:40.431444 21919 raft_consensus.cc:2454] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:40.431603 22445 raft_consensus.cc:2454] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:40.431846 22117 leader_election.cc:302] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:40.432121 22690 raft_consensus.cc:2781] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:40.432233 22690 raft_consensus.cc:482] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:40.432304 22690 raft_consensus.cc:3037] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.434295 22690 raft_consensus.cc:504] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.434859 22690 leader_election.cc:288] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:40.435223 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c891f8b1e54990b803493f459151ca" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:40.435413 21919 raft_consensus.cc:3037] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.435488 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c891f8b1e54990b803493f459151ca" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:40.435683 22445 raft_consensus.cc:3037] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.438280 21919 raft_consensus.cc:2454] T 81c891f8b1e54990b803493f459151ca P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:40.438475 22445 raft_consensus.cc:2454] T 81c891f8b1e54990b803493f459151ca P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:40.438679 22117 leader_election.cc:302] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:40.438951 22690 raft_consensus.cc:2781] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:40.439308 22690 raft_consensus.cc:686] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Running, Role: LEADER
I20240502 14:14:40.439607 22690 consensus_queue.cc:237] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.442066 21777 catalog_manager.cc:5484] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:40.466437 22935 raft_consensus.cc:482] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:40.466600 22935 raft_consensus.cc:504] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.467280 22935 leader_election.cc:288] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:40.467612 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51eb2447a234f44b89daf1af6d1ea87" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:40.467837 22314 raft_consensus.cc:2454] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:40.467788 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51eb2447a234f44b89daf1af6d1ea87" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:40.467998 22053 raft_consensus.cc:2454] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:40.468221 22513 leader_election.cc:302] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:40.468469 22935 raft_consensus.cc:2781] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:40.468575 22935 raft_consensus.cc:482] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:40.468662 22935 raft_consensus.cc:3037] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.470299 22935 raft_consensus.cc:504] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.470758 22935 leader_election.cc:288] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:40.471050 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51eb2447a234f44b89daf1af6d1ea87" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:40.471246 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51eb2447a234f44b89daf1af6d1ea87" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:40.471344 22314 raft_consensus.cc:3037] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.471434 22053 raft_consensus.cc:3037] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.473281 22314 raft_consensus.cc:2454] T a51eb2447a234f44b89daf1af6d1ea87 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:40.473603 22513 leader_election.cc:302] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:40.473853 22935 raft_consensus.cc:2781] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:40.474146 22053 raft_consensus.cc:2454] T a51eb2447a234f44b89daf1af6d1ea87 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:40.474503 22935 raft_consensus.cc:686] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:40.474961 22935 consensus_queue.cc:237] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:40.477454 21777 catalog_manager.cc:5484] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:40.643213 23011 raft_consensus.cc:482] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:40.643399 23011 raft_consensus.cc:504] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.644024 23011 leader_election.cc:288] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:40.644392 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2727235f406a456cbff925bda1f1cddb" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:40.644623 22184 raft_consensus.cc:2454] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:40.644616 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2727235f406a456cbff925bda1f1cddb" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:40.644836 21920 raft_consensus.cc:2454] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:40.645057 22381 leader_election.cc:302] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:40.645295 23011 raft_consensus.cc:2781] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:40.645395 23011 raft_consensus.cc:482] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:40.645465 23011 raft_consensus.cc:3037] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.647341 23011 raft_consensus.cc:504] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.647883 23011 leader_election.cc:288] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Requested vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:40.648182 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2727235f406a456cbff925bda1f1cddb" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:40.648370 22184 raft_consensus.cc:3037] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.648376 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2727235f406a456cbff925bda1f1cddb" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:40.648555 21920 raft_consensus.cc:3037] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.651154 21920 raft_consensus.cc:2454] T 2727235f406a456cbff925bda1f1cddb P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:40.651154 22184 raft_consensus.cc:2454] T 2727235f406a456cbff925bda1f1cddb P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:40.651505 22381 leader_election.cc:302] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:40.651788 23011 raft_consensus.cc:2781] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:40.651919 23011 raft_consensus.cc:686] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [term 1 LEADER]: Becoming Leader. State: Replica: fe26ecc72bed46179803517c549c5673, State: Running, Role: LEADER
I20240502 14:14:40.652166 23011 consensus_queue.cc:237] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:40.654599 21777 catalog_manager.cc:5484] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26ecc72bed46179803517c549c5673 (127.21.58.69). New cstate: current_term: 1 leader_uuid: "fe26ecc72bed46179803517c549c5673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:40.902910 22690 consensus_queue.cc:1035] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.910065 23042 consensus_queue.cc:1035] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.970328 22935 consensus_queue.cc:1035] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.977342 23044 consensus_queue.cc:1035] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:41.008255 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:41.008515 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:41.015483 23047 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:14:41.081894 23011 consensus_queue.cc:1035] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.098661 23011 consensus_queue.cc:1035] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 301.817 ms
time per row: 0.150909 ms
I20240502 14:14:41.450901 21777 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:41.459496 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 4d58f9f4a8b44df481f4093bcb7fe2d7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.459640 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 4d58f9f4a8b44df481f4093bcb7fe2d7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.459663 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 6820fc7861c9495388ac1458eb4956c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.459960 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d58f9f4a8b44df481f4093bcb7fe2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.460021 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d58f9f4a8b44df481f4093bcb7fe2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.460429 22163 tablet_service.cc:1467] Processing CreateTablet for tablet 44890bee90ca4ea0aa16acbd2f3dfcf2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.460665 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6820fc7861c9495388ac1458eb4956c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.460819 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44890bee90ca4ea0aa16acbd2f3dfcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.463711 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 44890bee90ca4ea0aa16acbd2f3dfcf2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.464133 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44890bee90ca4ea0aa16acbd2f3dfcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.464231 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 44890bee90ca4ea0aa16acbd2f3dfcf2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.464633 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44890bee90ca4ea0aa16acbd2f3dfcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.465924 22032 tablet_service.cc:1467] Processing CreateTablet for tablet 4d58f9f4a8b44df481f4093bcb7fe2d7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.466320 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d58f9f4a8b44df481f4093bcb7fe2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.471602 22557 tablet_service.cc:1467] Processing CreateTablet for tablet 6820fc7861c9495388ac1458eb4956c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.471935 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6820fc7861c9495388ac1458eb4956c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.475687 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 6820fc7861c9495388ac1458eb4956c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ece2f2dbf87549e8968b07cd90286bd5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.476095 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6820fc7861c9495388ac1458eb4956c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.485569 23123 tablet_bootstrap.cc:492] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:41.486104 23122 tablet_bootstrap.cc:492] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:41.486413 23120 tablet_bootstrap.cc:492] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:41.486860 23121 tablet_bootstrap.cc:492] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:41.488386 23120 tablet_bootstrap.cc:654] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.488626 23121 tablet_bootstrap.cc:654] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.489084 23123 tablet_bootstrap.cc:654] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.490175 23122 tablet_bootstrap.cc:654] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.504734 23123 tablet_bootstrap.cc:492] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:41.504971 23123 ts_tablet_manager.cc:1367] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20240502 14:14:41.505025 23128 tablet_bootstrap.cc:492] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:41.506023 23123 raft_consensus.cc:348] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.506457 23123 raft_consensus.cc:374] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.506698 23123 raft_consensus.cc:729] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.507853 23128 tablet_bootstrap.cc:654] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.508664 23119 tablet_bootstrap.cc:492] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:41.507997 23123 consensus_queue.cc:260] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.514612 23119 tablet_bootstrap.cc:654] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.515177 23123 ts_tablet_manager.cc:1398] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:14:41.517529 23122 tablet_bootstrap.cc:492] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:41.517724 23122 ts_tablet_manager.cc:1367] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.004s
I20240502 14:14:41.518479 23122 raft_consensus.cc:348] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.518740 23122 raft_consensus.cc:374] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.518967 23122 raft_consensus.cc:729] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.519546 23122 consensus_queue.cc:260] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.520717 23122 ts_tablet_manager.cc:1398] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:41.522089 23128 tablet_bootstrap.cc:492] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:41.522269 23128 ts_tablet_manager.cc:1367] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240502 14:14:41.523514 23128 raft_consensus.cc:348] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.524128 23128 raft_consensus.cc:374] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.524296 23128 raft_consensus.cc:729] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.524545 23121 tablet_bootstrap.cc:492] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:41.524691 23121 ts_tablet_manager.cc:1367] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.016s
I20240502 14:14:41.524682 23128 consensus_queue.cc:260] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.525475 23128 ts_tablet_manager.cc:1398] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:41.525383 23121 raft_consensus.cc:348] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.525549 23121 raft_consensus.cc:374] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.525637 23121 raft_consensus.cc:729] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.525823 23121 consensus_queue.cc:260] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.526456 23121 ts_tablet_manager.cc:1398] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:41.526810 23121 tablet_bootstrap.cc:492] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:41.528311 23121 tablet_bootstrap.cc:654] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.529986 23119 tablet_bootstrap.cc:492] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:41.530122 23119 ts_tablet_manager.cc:1367] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.005s
I20240502 14:14:41.530383 23120 tablet_bootstrap.cc:492] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:41.530558 23120 ts_tablet_manager.cc:1367] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.015s
I20240502 14:14:41.530951 23119 raft_consensus.cc:348] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.531177 23119 raft_consensus.cc:374] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.531275 23119 raft_consensus.cc:729] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.531309 23120 raft_consensus.cc:348] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.531518 23120 raft_consensus.cc:374] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.531633 23120 raft_consensus.cc:729] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.531607 23119 consensus_queue.cc:260] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.532008 23120 consensus_queue.cc:260] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.532220 23119 ts_tablet_manager.cc:1398] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:41.532665 23120 ts_tablet_manager.cc:1398] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:41.532982 23120 tablet_bootstrap.cc:492] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:41.533680 23119 tablet_bootstrap.cc:492] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:41.534474 23120 tablet_bootstrap.cc:654] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.535055 23119 tablet_bootstrap.cc:654] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.539667 23121 tablet_bootstrap.cc:492] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:41.539839 23121 ts_tablet_manager.cc:1367] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 14:14:41.540671 23121 raft_consensus.cc:348] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.540927 23121 raft_consensus.cc:374] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.541013 23121 raft_consensus.cc:729] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.541260 23121 consensus_queue.cc:260] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.542536 23121 ts_tablet_manager.cc:1398] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:41.547437 23119 tablet_bootstrap.cc:492] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:41.547437 23120 tablet_bootstrap.cc:492] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:41.547595 23119 ts_tablet_manager.cc:1367] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240502 14:14:41.547608 23120 ts_tablet_manager.cc:1367] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20240502 14:14:41.548259 23120 raft_consensus.cc:348] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.548342 23119 raft_consensus.cc:348] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.548513 23120 raft_consensus.cc:374] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.548615 23120 raft_consensus.cc:729] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.548624 23119 raft_consensus.cc:374] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.548686 23119 raft_consensus.cc:729] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.548830 23120 consensus_queue.cc:260] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.548954 23119 consensus_queue.cc:260] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.549324 23120 ts_tablet_manager.cc:1398] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:41.549379 23119 ts_tablet_manager.cc:1398] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:41.584415 23004 raft_consensus.cc:482] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.584547 23004 raft_consensus.cc:504] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.585249 23004 leader_election.cc:288] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:41.585655 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44890bee90ca4ea0aa16acbd2f3dfcf2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:41.585695 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44890bee90ca4ea0aa16acbd2f3dfcf2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:41.585870 22183 raft_consensus.cc:2454] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:41.585948 21919 raft_consensus.cc:2454] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:41.586261 22381 leader_election.cc:302] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:41.586303 22735 raft_consensus.cc:482] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.586457 22735 raft_consensus.cc:504] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.586730 23011 raft_consensus.cc:2781] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.586844 23011 raft_consensus.cc:482] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.586926 23011 raft_consensus.cc:3037] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.587064 22735 leader_election.cc:288] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:41.587581 22577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820fc7861c9495388ac1458eb4956c6" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:41.587723 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820fc7861c9495388ac1458eb4956c6" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:41.587781 22577 raft_consensus.cc:2454] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:41.587921 22314 raft_consensus.cc:2454] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:41.588153 21987 leader_election.cc:302] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:41.588483 22735 raft_consensus.cc:2781] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.588605 22735 raft_consensus.cc:482] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.588724 22735 raft_consensus.cc:3037] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.589350 23011 raft_consensus.cc:504] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.589790 23011 leader_election.cc:288] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Requested vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:41.590111 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44890bee90ca4ea0aa16acbd2f3dfcf2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:41.590268 22183 raft_consensus.cc:3037] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.590225 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44890bee90ca4ea0aa16acbd2f3dfcf2" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:41.590376 21920 raft_consensus.cc:3037] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.590651 22735 raft_consensus.cc:504] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.591243 22735 leader_election.cc:288] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:41.591616 22577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820fc7861c9495388ac1458eb4956c6" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:41.591781 22577 raft_consensus.cc:3037] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.592372 21920 raft_consensus.cc:2454] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:41.592721 22379 leader_election.cc:302] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:41.592943 23011 raft_consensus.cc:2781] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.592978 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820fc7861c9495388ac1458eb4956c6" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:41.593060 23011 raft_consensus.cc:686] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [term 1 LEADER]: Becoming Leader. State: Replica: fe26ecc72bed46179803517c549c5673, State: Running, Role: LEADER
I20240502 14:14:41.593125 22314 raft_consensus.cc:3037] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.593258 23011 consensus_queue.cc:237] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:41.594225 22577 raft_consensus.cc:2454] T 6820fc7861c9495388ac1458eb4956c6 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:41.594616 21987 leader_election.cc:302] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:41.594920 22735 raft_consensus.cc:2781] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.595047 22183 raft_consensus.cc:2454] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:41.595741 22314 raft_consensus.cc:2454] T 6820fc7861c9495388ac1458eb4956c6 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:41.596294 22735 raft_consensus.cc:686] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [term 1 LEADER]: Becoming Leader. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Running, Role: LEADER
I20240502 14:14:41.596101 21779 catalog_manager.cc:5484] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26ecc72bed46179803517c549c5673 (127.21.58.69). New cstate: current_term: 1 leader_uuid: "fe26ecc72bed46179803517c549c5673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:41.596570 22735 consensus_queue.cc:237] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:41.599117 21775 catalog_manager.cc:5484] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26b209fc9842d28d31c6243c869061 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "fe26b209fc9842d28d31c6243c869061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:41.694914 22735 raft_consensus.cc:482] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.695075 22735 raft_consensus.cc:504] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.695596 22735 leader_election.cc:288] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:41.695937 22577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d58f9f4a8b44df481f4093bcb7fe2d7" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:41.695948 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d58f9f4a8b44df481f4093bcb7fe2d7" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:41.696110 22577 raft_consensus.cc:2454] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:41.696123 22314 raft_consensus.cc:2454] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:41.696449 21987 leader_election.cc:302] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:41.696672 22735 raft_consensus.cc:2781] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.696791 22735 raft_consensus.cc:482] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.696892 22735 raft_consensus.cc:3037] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.698630 22735 raft_consensus.cc:504] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.699148 22735 leader_election.cc:288] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Requested vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:41.699457 22577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d58f9f4a8b44df481f4093bcb7fe2d7" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:41.699517 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d58f9f4a8b44df481f4093bcb7fe2d7" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:41.699587 22577 raft_consensus.cc:3037] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.699700 22314 raft_consensus.cc:3037] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.701241 22577 raft_consensus.cc:2454] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:41.701380 22314 raft_consensus.cc:2454] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:41.701689 21987 leader_election.cc:302] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:41.701970 22735 raft_consensus.cc:2781] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.702090 22735 raft_consensus.cc:686] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [term 1 LEADER]: Becoming Leader. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Running, Role: LEADER
I20240502 14:14:41.702343 22735 consensus_queue.cc:237] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:41.704834 21775 catalog_manager.cc:5484] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26b209fc9842d28d31c6243c869061 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "fe26b209fc9842d28d31c6243c869061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:42.004179 23011 consensus_queue.cc:1035] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:42.010329 23011 consensus_queue.cc:1035] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:42.047278 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:42.047631 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:42.048663 22735 consensus_queue.cc:1035] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:42.058454 23138 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:42.059974 23135 consensus_queue.cc:1035] T 6820fc7861c9495388ac1458eb4956c6 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240502 14:14:42.141285 23135 consensus_queue.cc:1035] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:42.148072 23135 consensus_queue.cc:1035] T 4d58f9f4a8b44df481f4093bcb7fe2d7 P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:42.342247 21987 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.70:40085, user_credentials={real_user=slave}} blocked reactor thread for 74106.2us
INSERT report
rows total: 2000
time total: 248.502 ms
time per row: 0.124251 ms
I20240502 14:14:42.453789 21779 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:42.461892 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 161ee37964b64237b85db9e5f298a9d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.462054 22163 tablet_service.cc:1467] Processing CreateTablet for tablet 161ee37964b64237b85db9e5f298a9d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.462347 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161ee37964b64237b85db9e5f298a9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.462460 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161ee37964b64237b85db9e5f298a9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.462512 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 161ee37964b64237b85db9e5f298a9d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.462926 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161ee37964b64237b85db9e5f298a9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.463734 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 15f020ceb22f488991c3f0b36e3d9bf2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.464126 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f020ceb22f488991c3f0b36e3d9bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.467576 22032 tablet_service.cc:1467] Processing CreateTablet for tablet 15f020ceb22f488991c3f0b36e3d9bf2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.467972 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f020ceb22f488991c3f0b36e3d9bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.468024 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 15f020ceb22f488991c3f0b36e3d9bf2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.468403 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f020ceb22f488991c3f0b36e3d9bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.472879 22164 tablet_service.cc:1467] Processing CreateTablet for tablet 27d0e5f7218146ba898af46e0caea1e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.473281 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d0e5f7218146ba898af46e0caea1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.473431 21900 tablet_service.cc:1467] Processing CreateTablet for tablet 27d0e5f7218146ba898af46e0caea1e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.473770 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d0e5f7218146ba898af46e0caea1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.476161 23204 tablet_bootstrap.cc:492] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:42.479303 23204 tablet_bootstrap.cc:654] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.482043 22425 tablet_service.cc:1467] Processing CreateTablet for tablet 27d0e5f7218146ba898af46e0caea1e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=11fd69574943448394a766a5ddfd203a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.482370 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d0e5f7218146ba898af46e0caea1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.495618 23209 tablet_bootstrap.cc:492] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:42.496331 23210 tablet_bootstrap.cc:492] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:42.497087 23205 tablet_bootstrap.cc:492] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:42.498613 23209 tablet_bootstrap.cc:654] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.499065 23205 tablet_bootstrap.cc:654] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.499711 23208 tablet_bootstrap.cc:492] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:42.500360 23210 tablet_bootstrap.cc:654] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.500919 23204 tablet_bootstrap.cc:492] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:42.501108 23204 ts_tablet_manager.cc:1367] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.004s
I20240502 14:14:42.501755 23208 tablet_bootstrap.cc:654] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.501936 23204 raft_consensus.cc:348] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.502225 23206 tablet_bootstrap.cc:492] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:42.502235 23204 raft_consensus.cc:374] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.502327 23204 raft_consensus.cc:729] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.502673 23204 consensus_queue.cc:260] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.504693 23205 tablet_bootstrap.cc:492] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:42.504881 23205 ts_tablet_manager.cc:1367] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 14:14:42.505098 23206 tablet_bootstrap.cc:654] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.505908 23205 raft_consensus.cc:348] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.506142 23205 raft_consensus.cc:374] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.506230 23205 raft_consensus.cc:729] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.506464 23205 consensus_queue.cc:260] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.507228 23205 ts_tablet_manager.cc:1398] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 14:14:42.508399 23209 tablet_bootstrap.cc:492] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:42.508577 23209 ts_tablet_manager.cc:1367] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240502 14:14:42.509337 23209 raft_consensus.cc:348] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.509598 23209 raft_consensus.cc:374] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.509728 23209 raft_consensus.cc:729] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.509899 23208 tablet_bootstrap.cc:492] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:42.509896 23204 ts_tablet_manager.cc:1398] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 14:14:42.510074 23208 ts_tablet_manager.cc:1367] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:14:42.510025 23209 consensus_queue.cc:260] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.510596 23209 ts_tablet_manager.cc:1398] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:42.511022 23209 tablet_bootstrap.cc:492] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:42.512544 23209 tablet_bootstrap.cc:654] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.514356 23208 raft_consensus.cc:348] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.514657 23206 tablet_bootstrap.cc:492] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:42.514685 23208 raft_consensus.cc:374] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.514854 23206 ts_tablet_manager.cc:1367] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:14:42.515689 23206 raft_consensus.cc:348] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.515846 23206 raft_consensus.cc:374] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.515935 23208 raft_consensus.cc:729] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.515997 23206 raft_consensus.cc:729] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.516117 23209 tablet_bootstrap.cc:492] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:42.516279 23209 ts_tablet_manager.cc:1367] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:14:42.516249 23208 consensus_queue.cc:260] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.516371 23206 consensus_queue.cc:260] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.516786 23208 ts_tablet_manager.cc:1398] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:14:42.516855 23206 ts_tablet_manager.cc:1398] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:42.517064 23208 tablet_bootstrap.cc:492] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:42.517079 23206 tablet_bootstrap.cc:492] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:42.518074 23210 tablet_bootstrap.cc:492] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:42.518267 23210 ts_tablet_manager.cc:1367] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240502 14:14:42.520108 23210 raft_consensus.cc:348] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.520364 23210 raft_consensus.cc:374] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.520484 23210 raft_consensus.cc:729] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.520895 23210 consensus_queue.cc:260] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.521034 23208 tablet_bootstrap.cc:654] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.521507 23210 ts_tablet_manager.cc:1398] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:14:42.521716 23206 tablet_bootstrap.cc:654] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.522337 23209 raft_consensus.cc:348] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.522610 23209 raft_consensus.cc:374] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.522729 23209 raft_consensus.cc:729] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.523013 23209 consensus_queue.cc:260] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.523821 23209 ts_tablet_manager.cc:1398] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:42.525252 23206 tablet_bootstrap.cc:492] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:42.525661 23206 ts_tablet_manager.cc:1367] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 14:14:42.526335 23206 raft_consensus.cc:348] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.526736 23206 raft_consensus.cc:374] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.526845 23206 raft_consensus.cc:729] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.527096 23206 consensus_queue.cc:260] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.527577 23208 tablet_bootstrap.cc:492] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:42.527704 23206 ts_tablet_manager.cc:1398] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:42.527783 23208 ts_tablet_manager.cc:1367] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:14:42.528496 23208 raft_consensus.cc:348] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.528862 23208 raft_consensus.cc:374] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.528959 23208 raft_consensus.cc:729] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.529187 23208 consensus_queue.cc:260] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.529796 23208 ts_tablet_manager.cc:1398] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:42.652956 23042 raft_consensus.cc:482] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:42.653138 23042 raft_consensus.cc:504] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.653726 23042 leader_election.cc:288] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:42.654148 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161ee37964b64237b85db9e5f298a9d8" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:42.654341 21920 raft_consensus.cc:2454] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:42.654402 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161ee37964b64237b85db9e5f298a9d8" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:42.654623 22446 raft_consensus.cc:2454] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:42.654757 22117 leader_election.cc:302] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:42.655042 23042 raft_consensus.cc:2781] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:42.655172 23042 raft_consensus.cc:482] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:42.655252 23042 raft_consensus.cc:3037] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.657173 23042 raft_consensus.cc:504] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.657590 22935 raft_consensus.cc:482] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:42.657739 23042 leader_election.cc:288] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:42.657756 22935 raft_consensus.cc:504] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.658051 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161ee37964b64237b85db9e5f298a9d8" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:42.658216 21920 raft_consensus.cc:3037] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.658371 22935 leader_election.cc:288] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:42.658717 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f020ceb22f488991c3f0b36e3d9bf2" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:42.658957 22051 raft_consensus.cc:2454] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:42.658962 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f020ceb22f488991c3f0b36e3d9bf2" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:42.659013 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161ee37964b64237b85db9e5f298a9d8" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:42.659211 22446 raft_consensus.cc:3037] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.659219 22314 raft_consensus.cc:2454] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:42.659407 22511 leader_election.cc:302] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:42.659653 22935 raft_consensus.cc:2781] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:42.659775 22935 raft_consensus.cc:482] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:42.659855 22935 raft_consensus.cc:3037] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.660607 21920 raft_consensus.cc:2454] T 161ee37964b64237b85db9e5f298a9d8 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:42.660953 22117 leader_election.cc:302] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:42.661195 23042 raft_consensus.cc:2781] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:42.661634 23042 raft_consensus.cc:686] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Running, Role: LEADER
I20240502 14:14:42.661765 22446 raft_consensus.cc:2454] T 161ee37964b64237b85db9e5f298a9d8 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:42.662034 23042 consensus_queue.cc:237] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.662621 22935 raft_consensus.cc:504] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.663101 22935 leader_election.cc:288] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:42.663527 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f020ceb22f488991c3f0b36e3d9bf2" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:42.663687 22314 raft_consensus.cc:3037] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.664146 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f020ceb22f488991c3f0b36e3d9bf2" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:42.664330 22051 raft_consensus.cc:3037] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.666399 22314 raft_consensus.cc:2454] T 15f020ceb22f488991c3f0b36e3d9bf2 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:42.666759 22513 leader_election.cc:302] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:42.666996 22051 raft_consensus.cc:2454] T 15f020ceb22f488991c3f0b36e3d9bf2 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:42.667007 22935 raft_consensus.cc:2781] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:42.667447 22935 raft_consensus.cc:686] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:42.667733 22935 consensus_queue.cc:237] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:42.668040 21779 catalog_manager.cc:5484] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:42.670111 21771 catalog_manager.cc:5484] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:42.731653 22686 raft_consensus.cc:482] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:42.731813 22686 raft_consensus.cc:504] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.732446 22686 leader_election.cc:288] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:42.732735 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0e5f7218146ba898af46e0caea1e4" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:42.732950 22183 raft_consensus.cc:2454] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:42.732942 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0e5f7218146ba898af46e0caea1e4" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:42.733187 22446 raft_consensus.cc:2454] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:42.733325 21856 leader_election.cc:302] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:42.733582 22686 raft_consensus.cc:2781] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:42.733690 22686 raft_consensus.cc:482] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:42.733770 22686 raft_consensus.cc:3037] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.735572 22686 raft_consensus.cc:504] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.736373 22686 leader_election.cc:288] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Requested vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:42.736433 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0e5f7218146ba898af46e0caea1e4" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:42.736563 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0e5f7218146ba898af46e0caea1e4" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:42.736624 22183 raft_consensus.cc:3037] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.736747 22446 raft_consensus.cc:3037] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.738421 22183 raft_consensus.cc:2454] T 27d0e5f7218146ba898af46e0caea1e4 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:42.738751 21856 leader_election.cc:302] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:42.739159 22686 raft_consensus.cc:2781] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:42.739758 22446 raft_consensus.cc:2454] T 27d0e5f7218146ba898af46e0caea1e4 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:42.740525 22686 raft_consensus.cc:686] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Running, Role: LEADER
I20240502 14:14:42.741021 22686 consensus_queue.cc:237] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:42.743588 21771 catalog_manager.cc:5484] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "4b0a27c175a3442dac0cae3f5b505110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:43.078066 23226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:43.078310 23226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:43.081866 23042 consensus_queue.cc:1035] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.086861 23219 consensus_queue.cc:1035] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:43.089125 23226 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 14:14:43.138931 22686 consensus_queue.cc:1035] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.143576 22686 consensus_queue.cc:1035] T 27d0e5f7218146ba898af46e0caea1e4 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.192824 22935 consensus_queue.cc:1035] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.266913 22935 consensus_queue.cc:1035] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 273.705 ms
time per row: 0.136852 ms
I20240502 14:14:43.503093 21779 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:43.511538 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 699a11fb57824dfbbc243eee82a68777 (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.511827 22032 tablet_service.cc:1467] Processing CreateTablet for tablet 699a11fb57824dfbbc243eee82a68777 (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.511996 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a11fb57824dfbbc243eee82a68777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.512022 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 699a11fb57824dfbbc243eee82a68777 (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.512216 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a11fb57824dfbbc243eee82a68777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.512411 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a11fb57824dfbbc243eee82a68777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.515730 22294 tablet_service.cc:1467] Processing CreateTablet for tablet c8ecbdc3e6bd440dabe6a4107c8599ca (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.516019 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ecbdc3e6bd440dabe6a4107c8599ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.519268 22557 tablet_service.cc:1467] Processing CreateTablet for tablet c8ecbdc3e6bd440dabe6a4107c8599ca (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.519603 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ecbdc3e6bd440dabe6a4107c8599ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.519579 22033 tablet_service.cc:1467] Processing CreateTablet for tablet c8ecbdc3e6bd440dabe6a4107c8599ca (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.519905 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ecbdc3e6bd440dabe6a4107c8599ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.524737 22426 tablet_service.cc:1467] Processing CreateTablet for tablet bdecd74592ee4fa695f8d458954e1fcd (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.525269 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdecd74592ee4fa695f8d458954e1fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.528148 21901 tablet_service.cc:1467] Processing CreateTablet for tablet bdecd74592ee4fa695f8d458954e1fcd (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.528671 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdecd74592ee4fa695f8d458954e1fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.530372 23296 tablet_bootstrap.cc:492] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:43.530730 23297 tablet_bootstrap.cc:492] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:43.531699 22163 tablet_service.cc:1467] Processing CreateTablet for tablet bdecd74592ee4fa695f8d458954e1fcd (DEFAULT_TABLE table=rebalance_test_table_7 [id=651083055b3d4b279353f754abae5663]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.532111 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdecd74592ee4fa695f8d458954e1fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.532354 23297 tablet_bootstrap.cc:654] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.532513 23296 tablet_bootstrap.cc:654] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.533793 23295 tablet_bootstrap.cc:492] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:43.535863 23295 tablet_bootstrap.cc:654] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.538270 23296 tablet_bootstrap.cc:492] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:43.538466 23296 ts_tablet_manager.cc:1367] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 14:14:43.538796 23297 tablet_bootstrap.cc:492] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:43.539326 23296 raft_consensus.cc:348] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.539618 23296 raft_consensus.cc:374] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.539772 23296 raft_consensus.cc:729] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.540066 23297 ts_tablet_manager.cc:1367] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 14:14:43.540076 23296 consensus_queue.cc:260] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.540982 23297 raft_consensus.cc:348] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.541227 23297 raft_consensus.cc:374] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.541368 23296 ts_tablet_manager.cc:1398] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:43.541368 23297 raft_consensus.cc:729] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.541707 23296 tablet_bootstrap.cc:492] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:43.541584 23297 consensus_queue.cc:260] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.542124 23297 ts_tablet_manager.cc:1398] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:43.543278 23296 tablet_bootstrap.cc:654] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.543413 23297 tablet_bootstrap.cc:492] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:43.546247 23297 tablet_bootstrap.cc:654] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.548074 23301 tablet_bootstrap.cc:492] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:43.550220 23301 tablet_bootstrap.cc:654] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.551568 23304 tablet_bootstrap.cc:492] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:43.552070 23302 tablet_bootstrap.cc:492] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:43.553493 23304 tablet_bootstrap.cc:654] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.554152 23296 tablet_bootstrap.cc:492] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:43.554220 23301 tablet_bootstrap.cc:492] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:43.554365 23296 ts_tablet_manager.cc:1367] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20240502 14:14:43.554688 23301 ts_tablet_manager.cc:1367] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:14:43.554939 23302 tablet_bootstrap.cc:654] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.555147 23296 raft_consensus.cc:348] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.555447 23301 raft_consensus.cc:348] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.555650 23301 raft_consensus.cc:374] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.555755 23301 raft_consensus.cc:729] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.556519 23296 raft_consensus.cc:374] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.557238 23296 raft_consensus.cc:729] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.558143 23304 tablet_bootstrap.cc:492] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:43.558189 23295 tablet_bootstrap.cc:492] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:43.558305 23295 ts_tablet_manager.cc:1367] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20240502 14:14:43.558372 23304 ts_tablet_manager.cc:1367] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:43.558502 23297 tablet_bootstrap.cc:492] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:43.558693 23297 ts_tablet_manager.cc:1367] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20240502 14:14:43.559038 23295 raft_consensus.cc:348] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.559239 23295 raft_consensus.cc:374] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.559356 23295 raft_consensus.cc:729] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.559377 23304 raft_consensus.cc:348] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.559473 23297 raft_consensus.cc:348] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.559736 23304 raft_consensus.cc:374] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.559641 23295 consensus_queue.cc:260] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.559792 23297 raft_consensus.cc:374] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.559923 23304 raft_consensus.cc:729] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.559957 23297 raft_consensus.cc:729] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.560189 23304 consensus_queue.cc:260] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.560336 23295 ts_tablet_manager.cc:1398] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:43.560287 23297 consensus_queue.cc:260] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.560822 23297 ts_tablet_manager.cc:1398] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:43.561404 23295 tablet_bootstrap.cc:492] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:43.561239 23301 consensus_queue.cc:260] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.562760 23295 tablet_bootstrap.cc:654] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.562893 23296 consensus_queue.cc:260] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.563365 23304 ts_tablet_manager.cc:1398] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:43.564183 23296 ts_tablet_manager.cc:1398] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:14:43.564483 23301 ts_tablet_manager.cc:1398] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:14:43.569916 23302 tablet_bootstrap.cc:492] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:43.570050 23302 ts_tablet_manager.cc:1367] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240502 14:14:43.570676 23302 raft_consensus.cc:348] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.570806 23302 raft_consensus.cc:374] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.570901 23302 raft_consensus.cc:729] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.571120 23302 consensus_queue.cc:260] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.571640 23302 ts_tablet_manager.cc:1398] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:43.572903 23295 tablet_bootstrap.cc:492] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:43.573055 23295 ts_tablet_manager.cc:1367] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240502 14:14:43.573616 23295 raft_consensus.cc:348] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.573768 23295 raft_consensus.cc:374] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.573835 23295 raft_consensus.cc:729] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.574057 23295 consensus_queue.cc:260] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.574565 23295 ts_tablet_manager.cc:1398] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:43.585440 23278 raft_consensus.cc:482] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.585578 23278 raft_consensus.cc:504] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.586045 23278 leader_election.cc:288] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:43.586479 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ecbdc3e6bd440dabe6a4107c8599ca" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:43.586496 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ecbdc3e6bd440dabe6a4107c8599ca" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:43.586663 22314 raft_consensus.cc:2454] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:43.586679 22051 raft_consensus.cc:2454] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:43.587041 22511 leader_election.cc:302] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:43.587298 23278 raft_consensus.cc:2781] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.587386 23278 raft_consensus.cc:482] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.587502 23278 raft_consensus.cc:3037] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.589195 23278 raft_consensus.cc:504] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.589660 23278 leader_election.cc:288] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:43.590024 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ecbdc3e6bd440dabe6a4107c8599ca" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:43.590052 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ecbdc3e6bd440dabe6a4107c8599ca" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:43.590152 22314 raft_consensus.cc:3037] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.590245 22051 raft_consensus.cc:3037] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.591879 22314 raft_consensus.cc:2454] T c8ecbdc3e6bd440dabe6a4107c8599ca P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:43.592006 22051 raft_consensus.cc:2454] T c8ecbdc3e6bd440dabe6a4107c8599ca P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:43.592180 22513 leader_election.cc:302] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:43.592415 23278 raft_consensus.cc:2781] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.592563 23278 raft_consensus.cc:686] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:43.592799 23278 consensus_queue.cc:237] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:43.595350 21776 catalog_manager.cc:5484] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:43.634446 23278 raft_consensus.cc:482] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.634581 23278 raft_consensus.cc:504] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.635217 23278 leader_election.cc:288] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:43.635488 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a11fb57824dfbbc243eee82a68777" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:43.635618 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a11fb57824dfbbc243eee82a68777" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:43.635697 22314 raft_consensus.cc:2454] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:43.635864 22051 raft_consensus.cc:2454] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:43.636066 22513 leader_election.cc:302] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:43.636341 23278 raft_consensus.cc:2781] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.636435 23278 raft_consensus.cc:482] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.636528 23278 raft_consensus.cc:3037] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.638242 23278 raft_consensus.cc:504] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.638720 23278 leader_election.cc:288] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:43.639114 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a11fb57824dfbbc243eee82a68777" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:43.639231 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a11fb57824dfbbc243eee82a68777" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:43.639319 22314 raft_consensus.cc:3037] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.639417 22051 raft_consensus.cc:3037] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.641934 22314 raft_consensus.cc:2454] T 699a11fb57824dfbbc243eee82a68777 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:43.642254 22513 leader_election.cc:302] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:43.642511 23278 raft_consensus.cc:2781] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.642545 22051 raft_consensus.cc:2454] T 699a11fb57824dfbbc243eee82a68777 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:43.642666 23278 raft_consensus.cc:686] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:43.642937 23278 consensus_queue.cc:237] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:43.645378 21776 catalog_manager.cc:5484] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:43.889482 23042 raft_consensus.cc:482] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.889698 23042 raft_consensus.cc:504] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.890369 23042 leader_election.cc:288] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:43.890712 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecd74592ee4fa695f8d458954e1fcd" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:43.890830 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecd74592ee4fa695f8d458954e1fcd" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:43.890962 21920 raft_consensus.cc:2454] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:43.891105 22445 raft_consensus.cc:2454] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:43.891491 22117 leader_election.cc:302] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:43.891777 23042 raft_consensus.cc:2781] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.891862 23042 raft_consensus.cc:482] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.891969 23042 raft_consensus.cc:3037] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.893692 23042 raft_consensus.cc:504] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.894295 23042 leader_election.cc:288] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:43.894675 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecd74592ee4fa695f8d458954e1fcd" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:43.894742 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdecd74592ee4fa695f8d458954e1fcd" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:43.894855 21920 raft_consensus.cc:3037] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.894939 22445 raft_consensus.cc:3037] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.897107 21920 raft_consensus.cc:2454] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:43.897437 22117 leader_election.cc:302] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:43.897675 23042 raft_consensus.cc:2781] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.897863 22445 raft_consensus.cc:2454] T bdecd74592ee4fa695f8d458954e1fcd P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:43.897920 23042 raft_consensus.cc:686] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Running, Role: LEADER
I20240502 14:14:43.898170 23042 consensus_queue.cc:237] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } }
I20240502 14:14:43.901430 21776 catalog_manager.cc:5484] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:44.021557 23278 consensus_queue.cc:1035] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.026149 23278 consensus_queue.cc:1035] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.035707 23278 consensus_queue.cc:1035] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.044756 23278 consensus_queue.cc:1035] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:44.295004 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:44.295305 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:44.303392 23313 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 14:14:44.418380 23042 consensus_queue.cc:1035] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.492353 21920 raft_consensus.cc:1261] T bdecd74592ee4fa695f8d458954e1fcd P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Refusing update from remote peer 6adb94cf6e2145c6bc6033236a27c4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:44.504722 23357 consensus_queue.cc:1035] T bdecd74592ee4fa695f8d458954e1fcd P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.547677 23351 mvcc.cc:205] Tried to move back new op lower bound from 7023244429163139072 to 7023244426849386496. Current Snapshot: MvccSnapshot[applied={T|T < 7023244429163139072}]
I20240502 14:14:44.559191 23369 mvcc.cc:205] Tried to move back new op lower bound from 7023244429163139072 to 7023244426849386496. Current Snapshot: MvccSnapshot[applied={T|T < 7023244429163139072}]
INSERT report
rows total: 2000
time total: 245.452 ms
time per row: 0.122726 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:45.631580 21771 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:45.641492 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 572d3a690c944995a666e919b92d1814 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.641654 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 572d3a690c944995a666e919b92d1814 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.641877 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 572d3a690c944995a666e919b92d1814 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.641963 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572d3a690c944995a666e919b92d1814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.642050 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572d3a690c944995a666e919b92d1814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.642204 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572d3a690c944995a666e919b92d1814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.646128 22426 tablet_service.cc:1467] Processing CreateTablet for tablet e9b5501053f14bd29349ef1e3af43ec0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.646595 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b5501053f14bd29349ef1e3af43ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.647572 21901 tablet_service.cc:1467] Processing CreateTablet for tablet e9b5501053f14bd29349ef1e3af43ec0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.648058 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b5501053f14bd29349ef1e3af43ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.649608 22163 tablet_service.cc:1467] Processing CreateTablet for tablet e9b5501053f14bd29349ef1e3af43ec0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.650115 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b5501053f14bd29349ef1e3af43ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.652628 22425 tablet_service.cc:1467] Processing CreateTablet for tablet 331bc60cc22b4081b23e763e45bf581c (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.655236 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331bc60cc22b4081b23e763e45bf581c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.655988 23391 tablet_bootstrap.cc:492] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:45.657007 23389 tablet_bootstrap.cc:492] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:45.658270 23391 tablet_bootstrap.cc:654] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.658519 23389 tablet_bootstrap.cc:654] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.659929 23390 tablet_bootstrap.cc:492] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:45.661983 23390 tablet_bootstrap.cc:654] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.663627 23391 tablet_bootstrap.cc:492] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:45.663823 23391 ts_tablet_manager.cc:1367] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240502 14:14:45.664521 23391 raft_consensus.cc:348] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:45.664682 23391 raft_consensus.cc:374] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.664793 23391 raft_consensus.cc:729] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.665022 23391 consensus_queue.cc:260] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:45.665623 23391 ts_tablet_manager.cc:1398] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:45.666594 23390 tablet_bootstrap.cc:492] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:45.666769 23390 ts_tablet_manager.cc:1367] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:45.667511 23390 raft_consensus.cc:348] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:45.667775 23390 raft_consensus.cc:374] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.667923 23390 raft_consensus.cc:729] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.668299 23390 consensus_queue.cc:260] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:45.669737 23390 ts_tablet_manager.cc:1398] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:45.671198 22164 tablet_service.cc:1467] Processing CreateTablet for tablet 331bc60cc22b4081b23e763e45bf581c (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.671634 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331bc60cc22b4081b23e763e45bf581c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.681308 23395 tablet_bootstrap.cc:492] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:45.681912 21900 tablet_service.cc:1467] Processing CreateTablet for tablet 331bc60cc22b4081b23e763e45bf581c (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba59f78714f840b1ad065f0b9f697b4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.682350 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331bc60cc22b4081b23e763e45bf581c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.683297 23395 tablet_bootstrap.cc:654] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.691694 23395 tablet_bootstrap.cc:492] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:45.691928 23395 ts_tablet_manager.cc:1367] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 14:14:45.692216 23399 tablet_bootstrap.cc:492] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:45.692931 23398 tablet_bootstrap.cc:492] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:45.692799 23395 raft_consensus.cc:348] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.693024 23395 raft_consensus.cc:374] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.693166 23395 raft_consensus.cc:729] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.693432 23395 consensus_queue.cc:260] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.694335 23395 ts_tablet_manager.cc:1398] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:45.694743 23395 tablet_bootstrap.cc:492] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:45.695231 23398 tablet_bootstrap.cc:654] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.696005 23399 tablet_bootstrap.cc:654] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.697278 23398 tablet_bootstrap.cc:492] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:45.697464 23398 ts_tablet_manager.cc:1367] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:14:45.697782 23395 tablet_bootstrap.cc:654] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.698475 23389 tablet_bootstrap.cc:492] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:45.698688 23389 ts_tablet_manager.cc:1367] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.016s
I20240502 14:14:45.698693 23398 raft_consensus.cc:348] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.699509 23398 raft_consensus.cc:374] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.699640 23398 raft_consensus.cc:729] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.699801 23399 tablet_bootstrap.cc:492] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:45.699982 23399 ts_tablet_manager.cc:1367] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 14:14:45.699954 23398 consensus_queue.cc:260] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.700116 23395 tablet_bootstrap.cc:492] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:45.700259 23395 ts_tablet_manager.cc:1367] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 14:14:45.700572 23398 ts_tablet_manager.cc:1398] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:45.700852 23398 tablet_bootstrap.cc:492] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:45.701206 23395 raft_consensus.cc:348] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.701459 23395 raft_consensus.cc:374] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.701571 23395 raft_consensus.cc:729] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.701887 23395 consensus_queue.cc:260] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.702418 23389 raft_consensus.cc:348] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:45.701997 23399 raft_consensus.cc:348] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.702802 23399 raft_consensus.cc:374] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.702934 23399 raft_consensus.cc:729] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.703094 23389 raft_consensus.cc:374] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.703263 23398 tablet_bootstrap.cc:654] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.703358 23395 ts_tablet_manager.cc:1398] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:45.703795 23399 consensus_queue.cc:260] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.704391 23399 ts_tablet_manager.cc:1398] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:14:45.704010 23389 raft_consensus.cc:729] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.705179 23399 tablet_bootstrap.cc:492] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:45.705476 23389 consensus_queue.cc:260] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:45.707551 23398 tablet_bootstrap.cc:492] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:45.707739 23398 ts_tablet_manager.cc:1367] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 14:14:45.707981 23399 tablet_bootstrap.cc:654] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.708664 23398 raft_consensus.cc:348] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.708922 23398 raft_consensus.cc:374] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.709039 23398 raft_consensus.cc:729] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.709300 23398 consensus_queue.cc:260] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.709889 23398 ts_tablet_manager.cc:1398] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:45.710662 23389 ts_tablet_manager.cc:1398] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 14:14:45.711597 23399 tablet_bootstrap.cc:492] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:45.711755 23399 ts_tablet_manager.cc:1367] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 14:14:45.712383 23399 raft_consensus.cc:348] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.712548 23399 raft_consensus.cc:374] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.712625 23399 raft_consensus.cc:729] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.712841 23399 consensus_queue.cc:260] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.713367 23399 ts_tablet_manager.cc:1398] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:45.729799 23357 raft_consensus.cc:482] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:45.729977 23357 raft_consensus.cc:504] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.730660 23357 leader_election.cc:288] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:45.731042 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bc60cc22b4081b23e763e45bf581c" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:45.731256 22446 raft_consensus.cc:2454] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:45.731231 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bc60cc22b4081b23e763e45bf581c" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:45.731436 21920 raft_consensus.cc:2454] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 0.
I20240502 14:14:45.731760 22118 leader_election.cc:302] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb94cf6e2145c6bc6033236a27c4b8, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:45.732105 23357 raft_consensus.cc:2781] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:45.732201 23357 raft_consensus.cc:482] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:45.732292 23357 raft_consensus.cc:3037] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.735046 23357 raft_consensus.cc:504] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.735656 23357 leader_election.cc:288] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:45.736006 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bc60cc22b4081b23e763e45bf581c" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:45.736047 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bc60cc22b4081b23e763e45bf581c" candidate_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:45.736212 21920 raft_consensus.cc:3037] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.736212 22446 raft_consensus.cc:3037] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.738934 22446 raft_consensus.cc:2454] T 331bc60cc22b4081b23e763e45bf581c P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:45.738934 21920 raft_consensus.cc:2454] T 331bc60cc22b4081b23e763e45bf581c P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adb94cf6e2145c6bc6033236a27c4b8 in term 1.
I20240502 14:14:45.739351 22117 leader_election.cc:302] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:45.739795 23357 raft_consensus.cc:2781] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:45.740312 23357 raft_consensus.cc:686] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Running, Role: LEADER
I20240502 14:14:45.740767 23357 consensus_queue.cc:237] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.743391 21771 catalog_manager.cc:5484] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:45.904937 23384 raft_consensus.cc:482] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:45.905128 23384 raft_consensus.cc:504] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.905675 23384 leader_election.cc:288] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:45.906147 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5501053f14bd29349ef1e3af43ec0" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
I20240502 14:14:45.906234 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5501053f14bd29349ef1e3af43ec0" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:45.906411 21920 raft_consensus.cc:2454] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:45.906455 22183 raft_consensus.cc:2454] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 0.
I20240502 14:14:45.906828 22379 leader_election.cc:302] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:45.907179 23384 raft_consensus.cc:2781] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:45.907279 23384 raft_consensus.cc:482] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:45.907366 23384 raft_consensus.cc:3037] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.909225 23384 raft_consensus.cc:504] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.909695 23384 leader_election.cc:288] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:45.910111 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5501053f14bd29349ef1e3af43ec0" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110"
I20240502 14:14:45.910181 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5501053f14bd29349ef1e3af43ec0" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:45.910282 21920 raft_consensus.cc:3037] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.910375 22183 raft_consensus.cc:3037] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.912524 21920 raft_consensus.cc:2454] T e9b5501053f14bd29349ef1e3af43ec0 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:45.912742 22183 raft_consensus.cc:2454] T e9b5501053f14bd29349ef1e3af43ec0 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26ecc72bed46179803517c549c5673 in term 1.
I20240502 14:14:45.912914 22379 leader_election.cc:302] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:45.913256 23384 raft_consensus.cc:2781] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:45.913384 23384 raft_consensus.cc:686] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [term 1 LEADER]: Becoming Leader. State: Replica: fe26ecc72bed46179803517c549c5673, State: Running, Role: LEADER
I20240502 14:14:45.913640 23384 consensus_queue.cc:237] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:45.916498 21771 catalog_manager.cc:5484] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26ecc72bed46179803517c549c5673 (127.21.58.69). New cstate: current_term: 1 leader_uuid: "fe26ecc72bed46179803517c549c5673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:46.048368 22777 raft_consensus.cc:482] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:46.048544 22777 raft_consensus.cc:504] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:46.049167 22777 leader_election.cc:288] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:46.049566 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572d3a690c944995a666e919b92d1814" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:46.049662 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572d3a690c944995a666e919b92d1814" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:46.049783 22053 raft_consensus.cc:2454] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 0.
I20240502 14:14:46.049890 22576 raft_consensus.cc:2454] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 0.
I20240502 14:14:46.050294 22249 leader_election.cc:302] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:46.050581 22777 raft_consensus.cc:2781] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:46.050693 22777 raft_consensus.cc:482] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:46.050782 22777 raft_consensus.cc:3037] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.053298 22777 raft_consensus.cc:504] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:46.053835 22777 leader_election.cc:288] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 election: Requested vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:46.054303 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572d3a690c944995a666e919b92d1814" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:46.054374 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572d3a690c944995a666e919b92d1814" candidate_uuid: "597f2b4f10754298a4a428ccd386c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:46.054504 22053 raft_consensus.cc:3037] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.054561 22576 raft_consensus.cc:3037] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.056392 22053 raft_consensus.cc:2454] T 572d3a690c944995a666e919b92d1814 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 1.
I20240502 14:14:46.056408 22576 raft_consensus.cc:2454] T 572d3a690c944995a666e919b92d1814 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 597f2b4f10754298a4a428ccd386c7d3 in term 1.
I20240502 14:14:46.056725 22249 leader_election.cc:302] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:46.057076 22777 raft_consensus.cc:2781] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:46.057400 22777 raft_consensus.cc:686] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 LEADER]: Becoming Leader. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Running, Role: LEADER
I20240502 14:14:46.057655 22777 consensus_queue.cc:237] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:46.060452 21771 catalog_manager.cc:5484] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "597f2b4f10754298a4a428ccd386c7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:46.144764 23357 consensus_queue.cc:1035] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:46.149490 23405 consensus_queue.cc:1035] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:46.386628 23384 consensus_queue.cc:1035] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:46.393337 23396 consensus_queue.cc:1035] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:46.460433 23410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:46.460716 23410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:46.468156 23410 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:46.484257 22777 consensus_queue.cc:1035] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:46.497828 23408 consensus_queue.cc:1035] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 179.774 ms
time per row: 0.0898869 ms
-----------------------------------------------------
Suppressions used:
count bytes template
28 7248 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:47.779366 21776 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:47.788173 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 2e96ad6ded944064891578165429cc82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.788643 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e96ad6ded944064891578165429cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.788761 22295 tablet_service.cc:1467] Processing CreateTablet for tablet b8476984ef484ea9900f117ca2aed7de (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.789165 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8476984ef484ea9900f117ca2aed7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.789472 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 2e96ad6ded944064891578165429cc82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.789918 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e96ad6ded944064891578165429cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.790202 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 387d9f2366444337a89005fccbc87128 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.790620 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387d9f2366444337a89005fccbc87128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.791510 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 2e96ad6ded944064891578165429cc82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.791894 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e96ad6ded944064891578165429cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.794194 22163 tablet_service.cc:1467] Processing CreateTablet for tablet 387d9f2366444337a89005fccbc87128 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.794611 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387d9f2366444337a89005fccbc87128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.797165 22557 tablet_service.cc:1467] Processing CreateTablet for tablet b8476984ef484ea9900f117ca2aed7de (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.797505 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8476984ef484ea9900f117ca2aed7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.789492 22032 tablet_service.cc:1467] Processing CreateTablet for tablet b8476984ef484ea9900f117ca2aed7de (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.798763 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8476984ef484ea9900f117ca2aed7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.802217 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 387d9f2366444337a89005fccbc87128 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9e73645e75546fe897ced0452401c69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:47.802723 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387d9f2366444337a89005fccbc87128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:47.804963 23480 tablet_bootstrap.cc:492] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:47.807605 23480 tablet_bootstrap.cc:654] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.810441 23481 tablet_bootstrap.cc:492] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:47.813252 23481 tablet_bootstrap.cc:654] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.817327 23483 tablet_bootstrap.cc:492] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:47.819025 23483 tablet_bootstrap.cc:654] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.825367 23481 tablet_bootstrap.cc:492] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:47.825526 23481 ts_tablet_manager.cc:1367] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240502 14:14:47.827276 23481 raft_consensus.cc:348] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:47.827787 23481 raft_consensus.cc:374] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.827956 23481 raft_consensus.cc:729] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.828238 23481 consensus_queue.cc:260] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:47.830511 23481 ts_tablet_manager.cc:1398] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:47.831820 23483 tablet_bootstrap.cc:492] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:47.831993 23483 ts_tablet_manager.cc:1367] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20240502 14:14:47.832067 23487 tablet_bootstrap.cc:492] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:47.833135 23483 raft_consensus.cc:348] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:47.833406 23483 raft_consensus.cc:374] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.833534 23483 raft_consensus.cc:729] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.833957 23487 tablet_bootstrap.cc:654] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.833801 23483 consensus_queue.cc:260] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:47.834442 23483 ts_tablet_manager.cc:1398] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:47.834770 23483 tablet_bootstrap.cc:492] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:47.836284 23486 tablet_bootstrap.cc:492] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:47.837039 23483 tablet_bootstrap.cc:654] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.837527 23485 tablet_bootstrap.cc:492] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:47.838864 23486 tablet_bootstrap.cc:654] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.840690 23396 raft_consensus.cc:482] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:47.840827 23396 raft_consensus.cc:504] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:47.841396 23396 leader_election.cc:288] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171)
I20240502 14:14:47.842013 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d9f2366444337a89005fccbc87128" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
W20240502 14:14:47.843504 22381 leader_election.cc:341] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:47.844316 21919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d9f2366444337a89005fccbc87128" candidate_uuid: "fe26ecc72bed46179803517c549c5673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0a27c175a3442dac0cae3f5b505110" is_pre_election: true
W20240502 14:14:47.845029 22379 leader_election.cc:341] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:47.845168 22379 leader_election.cc:302] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe26ecc72bed46179803517c549c5673; no voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8
I20240502 14:14:47.845420 23396 raft_consensus.cc:2726] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:47.845728 23483 tablet_bootstrap.cc:492] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:47.845882 23485 tablet_bootstrap.cc:654] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.845935 23483 ts_tablet_manager.cc:1367] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.001s
I20240502 14:14:47.846691 23483 raft_consensus.cc:348] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.846911 23483 raft_consensus.cc:374] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.847005 23483 raft_consensus.cc:729] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.847326 23483 consensus_queue.cc:260] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.848068 23483 ts_tablet_manager.cc:1398] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:47.850657 23485 tablet_bootstrap.cc:492] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:47.850679 23480 tablet_bootstrap.cc:492] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:47.850821 23480 ts_tablet_manager.cc:1367] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.022s
I20240502 14:14:47.850893 23485 ts_tablet_manager.cc:1367] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:14:47.851812 23485 raft_consensus.cc:348] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:47.852085 23485 raft_consensus.cc:374] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.852219 23485 raft_consensus.cc:729] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.852483 23485 consensus_queue.cc:260] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:47.852895 23480 raft_consensus.cc:348] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.853120 23485 ts_tablet_manager.cc:1398] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:47.853176 23480 raft_consensus.cc:374] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.854010 23480 raft_consensus.cc:729] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.854255 23480 consensus_queue.cc:260] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.855581 23480 ts_tablet_manager.cc:1398] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:47.855840 23480 tablet_bootstrap.cc:492] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:47.858075 23480 tablet_bootstrap.cc:654] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.861101 23487 tablet_bootstrap.cc:492] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:47.861723 23487 ts_tablet_manager.cc:1367] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240502 14:14:47.863416 23487 raft_consensus.cc:348] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:47.863652 23487 raft_consensus.cc:374] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.863821 23487 raft_consensus.cc:729] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.864209 23480 tablet_bootstrap.cc:492] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:47.864219 23486 tablet_bootstrap.cc:492] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:47.864109 23487 consensus_queue.cc:260] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:47.864354 23480 ts_tablet_manager.cc:1367] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20240502 14:14:47.864356 23486 ts_tablet_manager.cc:1367] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.014s
I20240502 14:14:47.864745 23487 ts_tablet_manager.cc:1398] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:47.865027 23480 raft_consensus.cc:348] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:47.865068 23486 raft_consensus.cc:348] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:47.865238 23480 raft_consensus.cc:374] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.865286 23486 raft_consensus.cc:374] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.865329 23480 raft_consensus.cc:729] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.865403 23486 raft_consensus.cc:729] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.865547 23480 consensus_queue.cc:260] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:47.865660 23486 consensus_queue.cc:260] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:47.866173 23480 ts_tablet_manager.cc:1398] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:47.866864 23486 ts_tablet_manager.cc:1398] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:47.867197 23486 tablet_bootstrap.cc:492] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:47.868765 23486 tablet_bootstrap.cc:654] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:47.875869 23486 tablet_bootstrap.cc:492] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:47.875996 23486 ts_tablet_manager.cc:1367] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240502 14:14:47.876458 23486 raft_consensus.cc:348] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.876642 23486 raft_consensus.cc:374] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:47.876711 23486 raft_consensus.cc:729] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:47.876865 23486 consensus_queue.cc:260] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.877374 23486 ts_tablet_manager.cc:1398] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:14:47.901654 23354 raft_consensus.cc:482] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:47.901811 23354 raft_consensus.cc:504] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.902542 23354 leader_election.cc:288] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:47.902892 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8476984ef484ea9900f117ca2aed7de" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:47.902940 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8476984ef484ea9900f117ca2aed7de" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:47.903126 22315 raft_consensus.cc:2454] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:47.903136 22053 raft_consensus.cc:2454] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:47.903522 22511 leader_election.cc:302] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:47.903771 23354 raft_consensus.cc:2781] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:47.903863 23354 raft_consensus.cc:482] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:47.903932 23354 raft_consensus.cc:3037] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.943406 23354 raft_consensus.cc:504] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.944105 23354 leader_election.cc:288] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:47.944504 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8476984ef484ea9900f117ca2aed7de" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:47.944582 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8476984ef484ea9900f117ca2aed7de" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:47.944700 22315 raft_consensus.cc:3037] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.944797 22053 raft_consensus.cc:3037] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.947511 22315 raft_consensus.cc:2454] T b8476984ef484ea9900f117ca2aed7de P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:47.948029 22513 leader_election.cc:302] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:47.948355 23354 raft_consensus.cc:2781] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:47.948483 23354 raft_consensus.cc:686] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:47.948781 23354 consensus_queue.cc:237] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } }
I20240502 14:14:47.951423 21776 catalog_manager.cc:5484] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:48.037855 22053 raft_consensus.cc:2454] T b8476984ef484ea9900f117ca2aed7de P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:48.076444 23475 raft_consensus.cc:482] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:48.076606 23475 raft_consensus.cc:504] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:48.077291 23475 leader_election.cc:288] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:48.077631 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d9f2366444337a89005fccbc87128" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:48.077714 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d9f2366444337a89005fccbc87128" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:48.077843 22446 raft_consensus.cc:2454] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:48.077932 22183 raft_consensus.cc:2454] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:48.078291 21856 leader_election.cc:302] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:48.078552 23475 raft_consensus.cc:2781] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:48.078671 23475 raft_consensus.cc:482] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:48.078770 23475 raft_consensus.cc:3037] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.088408 23354 raft_consensus.cc:482] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:48.088573 23354 raft_consensus.cc:504] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:48.089184 23354 leader_election.cc:288] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:48.089557 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96ad6ded944064891578165429cc82" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:48.089872 22315 raft_consensus.cc:2454] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:48.090121 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96ad6ded944064891578165429cc82" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:48.090256 22513 leader_election.cc:302] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:48.090359 22053 raft_consensus.cc:2454] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:48.090564 23354 raft_consensus.cc:2781] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:48.090711 23354 raft_consensus.cc:482] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:48.090837 23354 raft_consensus.cc:3037] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.142813 23354 raft_consensus.cc:504] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:48.142967 23475 raft_consensus.cc:504] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:48.144101 23354 leader_election.cc:288] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657)
I20240502 14:14:48.144964 23475 leader_election.cc:288] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:48.145145 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d9f2366444337a89005fccbc87128" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:48.145612 22446 raft_consensus.cc:3037] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.146224 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96ad6ded944064891578165429cc82" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:48.146291 22183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387d9f2366444337a89005fccbc87128" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:48.146463 22053 raft_consensus.cc:3037] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.146492 22183 raft_consensus.cc:3037] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.149255 22053 raft_consensus.cc:2454] T 2e96ad6ded944064891578165429cc82 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:48.149255 22446 raft_consensus.cc:2454] T 387d9f2366444337a89005fccbc87128 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:48.149685 21855 leader_election.cc:302] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:48.144711 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96ad6ded944064891578165429cc82" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:48.149801 22511 leader_election.cc:302] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:48.149968 22315 raft_consensus.cc:3037] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.150063 23354 raft_consensus.cc:2781] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:48.150221 23354 raft_consensus.cc:686] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:48.150497 23354 consensus_queue.cc:237] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:48.152165 23475 raft_consensus.cc:2781] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:48.153314 23475 raft_consensus.cc:686] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Running, Role: LEADER
I20240502 14:14:48.153820 23475 consensus_queue.cc:237] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:48.154834 21776 catalog_manager.cc:5484] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:48.156673 21771 catalog_manager.cc:5484] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "4b0a27c175a3442dac0cae3f5b505110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:48.248942 22315 raft_consensus.cc:2454] T 2e96ad6ded944064891578165429cc82 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:48.248953 22183 raft_consensus.cc:2454] T 387d9f2366444337a89005fccbc87128 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:48.354962 23354 consensus_queue.cc:1035] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:48.363345 23495 consensus_queue.cc:1035] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:48.504492 23501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:48.504809 23501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:48.512331 23501 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 14:14:48.595530 23354 consensus_queue.cc:1035] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:48.673044 23495 consensus_queue.cc:1035] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:48.686517 23475 consensus_queue.cc:1035] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:48.738399 23475 consensus_queue.cc:1035] T 387d9f2366444337a89005fccbc87128 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 353.94 ms
time per row: 0.17697 ms
I20240502 14:14:49.022505 21772 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:49.034317 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae862f0ebab4b078f60aae9c4379b5d (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.034824 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae862f0ebab4b078f60aae9c4379b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.035331 22033 tablet_service.cc:1467] Processing CreateTablet for tablet a688ca99359c4f1ead97f159aa59e14c (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.035466 22294 tablet_service.cc:1467] Processing CreateTablet for tablet a688ca99359c4f1ead97f159aa59e14c (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.035759 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a688ca99359c4f1ead97f159aa59e14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.035930 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a688ca99359c4f1ead97f159aa59e14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.039546 22163 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae862f0ebab4b078f60aae9c4379b5d (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.039925 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae862f0ebab4b078f60aae9c4379b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.043650 22556 tablet_service.cc:1467] Processing CreateTablet for tablet a688ca99359c4f1ead97f159aa59e14c (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.044029 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a688ca99359c4f1ead97f159aa59e14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.048411 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae862f0ebab4b078f60aae9c4379b5d (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.048480 21900 tablet_service.cc:1467] Processing CreateTablet for tablet 925a4efd9c1040a4a1303ee999a5c995 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.048833 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae862f0ebab4b078f60aae9c4379b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.049036 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925a4efd9c1040a4a1303ee999a5c995. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.053790 22425 tablet_service.cc:1467] Processing CreateTablet for tablet 925a4efd9c1040a4a1303ee999a5c995 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.053819 22164 tablet_service.cc:1467] Processing CreateTablet for tablet 925a4efd9c1040a4a1303ee999a5c995 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3d7a8290d67c437da68f0af8129a6d0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.054167 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925a4efd9c1040a4a1303ee999a5c995. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.054167 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925a4efd9c1040a4a1303ee999a5c995. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.061748 23574 tablet_bootstrap.cc:492] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:49.063578 23574 tablet_bootstrap.cc:654] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.065196 23578 tablet_bootstrap.cc:492] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:49.068197 23576 tablet_bootstrap.cc:492] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:49.070281 23576 tablet_bootstrap.cc:654] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.073681 23575 tablet_bootstrap.cc:492] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:49.076243 23575 tablet_bootstrap.cc:654] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.081696 23580 tablet_bootstrap.cc:492] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:49.082120 23579 tablet_bootstrap.cc:492] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:49.083532 23580 tablet_bootstrap.cc:654] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.083923 23578 tablet_bootstrap.cc:654] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.084097 23579 tablet_bootstrap.cc:654] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.089970 23579 tablet_bootstrap.cc:492] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:49.090394 23579 ts_tablet_manager.cc:1367] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 14:14:49.090901 23576 tablet_bootstrap.cc:492] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:49.091147 23576 ts_tablet_manager.cc:1367] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240502 14:14:49.091400 23579 raft_consensus.cc:348] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.091574 23580 tablet_bootstrap.cc:492] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:49.091643 23579 raft_consensus.cc:374] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.091769 23575 tablet_bootstrap.cc:492] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:49.091769 23579 raft_consensus.cc:729] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.091935 23575 ts_tablet_manager.cc:1367] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20240502 14:14:49.092038 23579 consensus_queue.cc:260] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.092207 23576 raft_consensus.cc:348] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.092563 23576 raft_consensus.cc:374] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.092675 23579 ts_tablet_manager.cc:1398] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:49.092694 23576 raft_consensus.cc:729] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.092991 23576 consensus_queue.cc:260] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.093160 23579 tablet_bootstrap.cc:492] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:49.093325 23354 raft_consensus.cc:482] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.093446 23354 raft_consensus.cc:504] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.094108 23354 leader_election.cc:288] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:49.094652 23579 tablet_bootstrap.cc:654] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.095376 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a688ca99359c4f1ead97f159aa59e14c" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
W20240502 14:14:49.096287 22513 leader_election.cc:341] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:49.098227 23576 ts_tablet_manager.cc:1398] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:49.095049 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a688ca99359c4f1ead97f159aa59e14c" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:49.098474 23580 ts_tablet_manager.cc:1367] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.001s
W20240502 14:14:49.099021 22511 leader_election.cc:341] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:49.099162 22511 leader_election.cc:302] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a; no voters: 597f2b4f10754298a4a428ccd386c7d3, fe26b209fc9842d28d31c6243c869061
I20240502 14:14:49.099365 23580 raft_consensus.cc:348] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.099537 23580 raft_consensus.cc:374] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.099622 23580 raft_consensus.cc:729] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.099696 23354 raft_consensus.cc:2726] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:49.099845 23580 consensus_queue.cc:260] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.100520 23575 raft_consensus.cc:348] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.100769 23575 raft_consensus.cc:374] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.100862 23575 raft_consensus.cc:729] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.101125 23575 consensus_queue.cc:260] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.101892 23575 ts_tablet_manager.cc:1398] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 14:14:49.102463 23580 ts_tablet_manager.cc:1398] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:49.104274 23574 tablet_bootstrap.cc:492] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:49.104444 23574 ts_tablet_manager.cc:1367] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.006s
I20240502 14:14:49.105361 23574 raft_consensus.cc:348] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.105587 23574 raft_consensus.cc:374] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.105693 23574 raft_consensus.cc:729] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.106038 23574 consensus_queue.cc:260] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.107986 23574 ts_tablet_manager.cc:1398] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 14:14:49.108327 23574 tablet_bootstrap.cc:492] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:49.112869 23579 tablet_bootstrap.cc:492] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:49.113117 23579 ts_tablet_manager.cc:1367] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240502 14:14:49.114105 23579 raft_consensus.cc:348] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.114311 23579 raft_consensus.cc:374] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.114377 23579 raft_consensus.cc:729] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.114593 23579 consensus_queue.cc:260] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.115051 23579 ts_tablet_manager.cc:1398] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:49.123450 23574 tablet_bootstrap.cc:654] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.124008 23578 tablet_bootstrap.cc:492] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:49.124159 23578 ts_tablet_manager.cc:1367] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.006s
I20240502 14:14:49.125003 23578 raft_consensus.cc:348] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.125097 23574 tablet_bootstrap.cc:492] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:49.125170 23578 raft_consensus.cc:374] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.125226 23574 ts_tablet_manager.cc:1367] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20240502 14:14:49.125265 23578 raft_consensus.cc:729] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.125496 23578 consensus_queue.cc:260] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.125803 23574 raft_consensus.cc:348] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.125943 23578 ts_tablet_manager.cc:1398] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:49.126019 23574 raft_consensus.cc:374] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.126106 23574 raft_consensus.cc:729] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.126302 23578 tablet_bootstrap.cc:492] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:49.126348 23574 consensus_queue.cc:260] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.127476 23574 ts_tablet_manager.cc:1398] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:49.127846 23578 tablet_bootstrap.cc:654] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.129622 23578 tablet_bootstrap.cc:492] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:49.129827 23578 ts_tablet_manager.cc:1367] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:14:49.130410 23578 raft_consensus.cc:348] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.130569 23578 raft_consensus.cc:374] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.130688 23578 raft_consensus.cc:729] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.130985 23578 consensus_queue.cc:260] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.131613 23578 ts_tablet_manager.cc:1398] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:49.143257 23475 raft_consensus.cc:482] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.143429 23475 raft_consensus.cc:504] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.143918 23475 leader_election.cc:288] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:49.144285 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae862f0ebab4b078f60aae9c4379b5d" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:49.144376 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae862f0ebab4b078f60aae9c4379b5d" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:49.144464 22184 raft_consensus.cc:2454] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:49.144580 22446 raft_consensus.cc:2454] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:49.144845 21856 leader_election.cc:302] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:49.145107 23475 raft_consensus.cc:2781] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:49.145186 23475 raft_consensus.cc:482] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:49.145267 23475 raft_consensus.cc:3037] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.147187 23475 raft_consensus.cc:504] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.147691 23475 leader_election.cc:288] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Requested vote from peers 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989), fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619)
I20240502 14:14:49.148000 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae862f0ebab4b078f60aae9c4379b5d" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:49.148125 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae862f0ebab4b078f60aae9c4379b5d" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:49.148185 22184 raft_consensus.cc:3037] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.148296 22446 raft_consensus.cc:3037] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.150864 22184 raft_consensus.cc:2454] T 9ae862f0ebab4b078f60aae9c4379b5d P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:49.150964 22446 raft_consensus.cc:2454] T 9ae862f0ebab4b078f60aae9c4379b5d P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:49.151283 21856 leader_election.cc:302] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:49.151557 23475 raft_consensus.cc:2781] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:49.151705 23475 raft_consensus.cc:686] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Running, Role: LEADER
I20240502 14:14:49.151943 23475 consensus_queue.cc:237] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:49.154315 21771 catalog_manager.cc:5484] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "4b0a27c175a3442dac0cae3f5b505110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:49.235098 23500 raft_consensus.cc:482] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.235297 23500 raft_consensus.cc:504] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.235956 23500 leader_election.cc:288] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:49.236307 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a688ca99359c4f1ead97f159aa59e14c" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:49.236326 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a688ca99359c4f1ead97f159aa59e14c" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:49.236522 22315 raft_consensus.cc:2454] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:49.236531 22576 raft_consensus.cc:2454] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:49.236882 21989 leader_election.cc:302] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 597f2b4f10754298a4a428ccd386c7d3, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:49.237170 23500 raft_consensus.cc:2781] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:49.237265 23500 raft_consensus.cc:482] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:49.237356 23500 raft_consensus.cc:3037] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.239296 23500 raft_consensus.cc:504] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.239800 23500 leader_election.cc:288] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Requested vote from peers 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:49.240166 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a688ca99359c4f1ead97f159aa59e14c" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:49.240192 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a688ca99359c4f1ead97f159aa59e14c" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:49.240361 22576 raft_consensus.cc:3037] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.240357 22315 raft_consensus.cc:3037] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.242226 23475 raft_consensus.cc:482] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.242278 22576 raft_consensus.cc:2454] T a688ca99359c4f1ead97f159aa59e14c P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:49.242367 22315 raft_consensus.cc:2454] T a688ca99359c4f1ead97f159aa59e14c P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:49.242390 23475 raft_consensus.cc:504] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.242660 21987 leader_election.cc:302] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:49.242911 23500 raft_consensus.cc:2781] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:49.243230 23500 raft_consensus.cc:686] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [term 1 LEADER]: Becoming Leader. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Running, Role: LEADER
I20240502 14:14:49.243629 23475 leader_election.cc:288] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:49.243593 23500 consensus_queue.cc:237] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:49.243706 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925a4efd9c1040a4a1303ee999a5c995" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:49.243758 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925a4efd9c1040a4a1303ee999a5c995" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:49.243937 22184 raft_consensus.cc:2454] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:49.243969 22446 raft_consensus.cc:2454] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:49.244285 21856 leader_election.cc:302] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:49.244576 23475 raft_consensus.cc:2781] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:49.244753 23475 raft_consensus.cc:482] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:49.244868 23475 raft_consensus.cc:3037] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.247182 21779 catalog_manager.cc:5484] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26b209fc9842d28d31c6243c869061 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "fe26b209fc9842d28d31c6243c869061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:49.247926 23475 raft_consensus.cc:504] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.248515 23475 leader_election.cc:288] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:49.248899 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925a4efd9c1040a4a1303ee999a5c995" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:49.248961 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925a4efd9c1040a4a1303ee999a5c995" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:49.249079 22184 raft_consensus.cc:3037] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.249181 22446 raft_consensus.cc:3037] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.251101 22184 raft_consensus.cc:2454] T 925a4efd9c1040a4a1303ee999a5c995 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:49.251488 21856 leader_election.cc:302] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:49.251586 22446 raft_consensus.cc:2454] T 925a4efd9c1040a4a1303ee999a5c995 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:49.251765 23475 raft_consensus.cc:2781] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:49.251915 23475 raft_consensus.cc:686] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Running, Role: LEADER
I20240502 14:14:49.252166 23475 consensus_queue.cc:237] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } }
I20240502 14:14:49.254947 21779 catalog_manager.cc:5484] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "4b0a27c175a3442dac0cae3f5b505110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:49.623847 23592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:49.624166 23592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:49.632352 23592 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 14:14:49.672118 23500 consensus_queue.cc:1035] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.686479 23135 consensus_queue.cc:1035] T a688ca99359c4f1ead97f159aa59e14c P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.698685 23475 consensus_queue.cc:1035] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.707369 23475 consensus_queue.cc:1035] T 9ae862f0ebab4b078f60aae9c4379b5d P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.763114 23475 consensus_queue.cc:1035] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.784854 23541 consensus_queue.cc:1035] T 925a4efd9c1040a4a1303ee999a5c995 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:49.921276 21855 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.69:42619, user_credentials={real_user=slave}} blocked reactor thread for 93942.4us
INSERT report
rows total: 2000
time total: 292.037 ms
time per row: 0.146019 ms
I20240502 14:14:50.105715 21772 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54542:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:50.114953 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7d265043b247b4a708a68ade84ef78 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.115193 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7d265043b247b4a708a68ade84ef78 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.115305 22295 tablet_service.cc:1467] Processing CreateTablet for tablet e9034fb87e834c2fa59c6d7b549b19dc (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.115432 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7d265043b247b4a708a68ade84ef78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.115572 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7d265043b247b4a708a68ade84ef78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.116222 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9034fb87e834c2fa59c6d7b549b19dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.116127 22163 tablet_service.cc:1467] Processing CreateTablet for tablet dc3906b7604e4e109010948e9c45a947 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.116545 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3906b7604e4e109010948e9c45a947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.120182 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7d265043b247b4a708a68ade84ef78 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.120575 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7d265043b247b4a708a68ade84ef78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.123534 21901 tablet_service.cc:1467] Processing CreateTablet for tablet dc3906b7604e4e109010948e9c45a947 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.123947 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3906b7604e4e109010948e9c45a947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.128384 22557 tablet_service.cc:1467] Processing CreateTablet for tablet e9034fb87e834c2fa59c6d7b549b19dc (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.128576 22426 tablet_service.cc:1467] Processing CreateTablet for tablet dc3906b7604e4e109010948e9c45a947 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.128751 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9034fb87e834c2fa59c6d7b549b19dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.129151 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3906b7604e4e109010948e9c45a947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.131716 22032 tablet_service.cc:1467] Processing CreateTablet for tablet e9034fb87e834c2fa59c6d7b549b19dc (DEFAULT_TABLE table=rebalance_test_table_11 [id=1885a253772d480b8cf546fff5f5f071]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.132093 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9034fb87e834c2fa59c6d7b549b19dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.143978 23668 tablet_bootstrap.cc:492] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8: Bootstrap starting.
I20240502 14:14:50.144253 23669 tablet_bootstrap.cc:492] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:50.144942 23671 tablet_bootstrap.cc:492] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673: Bootstrap starting.
I20240502 14:14:50.148056 23669 tablet_bootstrap.cc:654] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.148357 23670 tablet_bootstrap.cc:492] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:50.149434 23667 tablet_bootstrap.cc:492] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110: Bootstrap starting.
I20240502 14:14:50.152063 23667 tablet_bootstrap.cc:654] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.152905 23671 tablet_bootstrap.cc:654] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.154032 23666 tablet_bootstrap.cc:492] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:50.154255 23668 tablet_bootstrap.cc:654] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.156193 23666 tablet_bootstrap.cc:654] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.156404 23670 tablet_bootstrap.cc:654] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.158138 23668 tablet_bootstrap.cc:492] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8: No bootstrap required, opened a new log
I20240502 14:14:50.158159 23667 tablet_bootstrap.cc:492] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110: No bootstrap required, opened a new log
I20240502 14:14:50.158237 23671 tablet_bootstrap.cc:492] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673: No bootstrap required, opened a new log
I20240502 14:14:50.158331 23668 ts_tablet_manager.cc:1367] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:14:50.158401 23667 ts_tablet_manager.cc:1367] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:14:50.158447 23671 ts_tablet_manager.cc:1367] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 14:14:50.159312 23668 raft_consensus.cc:348] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.159406 23671 raft_consensus.cc:348] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.159574 23668 raft_consensus.cc:374] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.159608 23671 raft_consensus.cc:374] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.159431 23667 raft_consensus.cc:348] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.159701 23671 raft_consensus.cc:729] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26ecc72bed46179803517c549c5673, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.159723 23668 raft_consensus.cc:729] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb94cf6e2145c6bc6033236a27c4b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.159809 23667 raft_consensus.cc:374] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.159993 23669 tablet_bootstrap.cc:492] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:50.159999 23667 raft_consensus.cc:729] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.159981 23668 consensus_queue.cc:260] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.160051 23671 consensus_queue.cc:260] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.160172 23669 ts_tablet_manager.cc:1367] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240502 14:14:50.160393 23667 consensus_queue.cc:260] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.160595 23671 ts_tablet_manager.cc:1398] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:50.160590 23668 ts_tablet_manager.cc:1398] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:50.161116 23667 ts_tablet_manager.cc:1398] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:50.162065 23666 tablet_bootstrap.cc:492] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:50.162240 23666 ts_tablet_manager.cc:1367] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 14:14:50.162829 23670 tablet_bootstrap.cc:492] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:50.162968 23670 ts_tablet_manager.cc:1367] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 14:14:50.163736 23670 raft_consensus.cc:348] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.163797 23666 raft_consensus.cc:348] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.164001 23666 raft_consensus.cc:374] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.164014 23670 raft_consensus.cc:374] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.164146 23670 raft_consensus.cc:729] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.164151 23666 raft_consensus.cc:729] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.164412 23666 consensus_queue.cc:260] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.164412 23670 consensus_queue.cc:260] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.165048 23666 ts_tablet_manager.cc:1398] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:50.165385 23666 tablet_bootstrap.cc:492] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3: Bootstrap starting.
I20240502 14:14:50.166079 23670 ts_tablet_manager.cc:1398] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:50.166512 23670 tablet_bootstrap.cc:492] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061: Bootstrap starting.
I20240502 14:14:50.167831 23666 tablet_bootstrap.cc:654] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.169054 23670 tablet_bootstrap.cc:654] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.173249 23669 raft_consensus.cc:348] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.173475 23669 raft_consensus.cc:374] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.173580 23669 raft_consensus.cc:729] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.173812 23669 consensus_queue.cc:260] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.175518 23669 ts_tablet_manager.cc:1398] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240502 14:14:50.175807 23669 tablet_bootstrap.cc:492] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a: Bootstrap starting.
I20240502 14:14:50.176646 23666 tablet_bootstrap.cc:492] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3: No bootstrap required, opened a new log
I20240502 14:14:50.176792 23666 ts_tablet_manager.cc:1367] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 14:14:50.177330 23666 raft_consensus.cc:348] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.177505 23666 raft_consensus.cc:374] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.177590 23666 raft_consensus.cc:729] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597f2b4f10754298a4a428ccd386c7d3, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.177815 23666 consensus_queue.cc:260] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.178252 23669 tablet_bootstrap.cc:654] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.178380 23666 ts_tablet_manager.cc:1398] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:50.190524 23669 tablet_bootstrap.cc:492] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a: No bootstrap required, opened a new log
I20240502 14:14:50.190680 23669 ts_tablet_manager.cc:1367] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20240502 14:14:50.191136 23670 tablet_bootstrap.cc:492] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061: No bootstrap required, opened a new log
I20240502 14:14:50.191285 23670 ts_tablet_manager.cc:1367] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.011s
I20240502 14:14:50.191394 23669 raft_consensus.cc:348] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.191609 23669 raft_consensus.cc:374] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.191704 23669 raft_consensus.cc:729] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.191920 23669 consensus_queue.cc:260] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.191946 23670 raft_consensus.cc:348] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.192121 23670 raft_consensus.cc:374] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.192185 23670 raft_consensus.cc:729] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.192385 23669 ts_tablet_manager.cc:1398] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:50.192400 23670 consensus_queue.cc:260] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.192932 23670 ts_tablet_manager.cc:1398] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:50.198194 23646 raft_consensus.cc:482] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.198307 23646 raft_consensus.cc:504] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.198886 23646 leader_election.cc:288] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:50.199340 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9034fb87e834c2fa59c6d7b549b19dc" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a" is_pre_election: true
I20240502 14:14:50.199529 22576 raft_consensus.cc:2454] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:50.199855 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9034fb87e834c2fa59c6d7b549b19dc" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:50.199913 21987 leader_election.cc:302] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:50.200040 22315 raft_consensus.cc:2454] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 0.
I20240502 14:14:50.200222 23646 raft_consensus.cc:2781] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.200338 23646 raft_consensus.cc:482] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.200430 23646 raft_consensus.cc:3037] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.202358 23646 raft_consensus.cc:504] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.202821 23646 leader_election.cc:288] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Requested vote from peers 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981), 491bfc422e6440749f54ffffdb717e3a (127.21.58.70:40085)
I20240502 14:14:50.203225 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9034fb87e834c2fa59c6d7b549b19dc" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491bfc422e6440749f54ffffdb717e3a"
I20240502 14:14:50.203415 22576 raft_consensus.cc:3037] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.203377 22315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9034fb87e834c2fa59c6d7b549b19dc" candidate_uuid: "fe26b209fc9842d28d31c6243c869061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:50.203545 22315 raft_consensus.cc:3037] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.206220 22576 raft_consensus.cc:2454] T e9034fb87e834c2fa59c6d7b549b19dc P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:50.206234 22315 raft_consensus.cc:2454] T e9034fb87e834c2fa59c6d7b549b19dc P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26b209fc9842d28d31c6243c869061 in term 1.
I20240502 14:14:50.206589 21987 leader_election.cc:302] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:50.206923 23646 raft_consensus.cc:2781] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.207067 23646 raft_consensus.cc:686] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [term 1 LEADER]: Becoming Leader. State: Replica: fe26b209fc9842d28d31c6243c869061, State: Running, Role: LEADER
I20240502 14:14:50.207422 23646 consensus_queue.cc:237] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } }
I20240502 14:14:50.210314 21777 catalog_manager.cc:5484] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26b209fc9842d28d31c6243c869061 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "fe26b209fc9842d28d31c6243c869061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:50.302254 23354 raft_consensus.cc:482] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.302440 23354 raft_consensus.cc:504] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.303166 23354 leader_election.cc:288] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:50.303537 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d265043b247b4a708a68ade84ef78" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061" is_pre_election: true
I20240502 14:14:50.303663 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d265043b247b4a708a68ade84ef78" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3" is_pre_election: true
I20240502 14:14:50.303792 22053 raft_consensus.cc:2454] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:50.303895 22314 raft_consensus.cc:2454] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 0.
I20240502 14:14:50.304268 22511 leader_election.cc:302] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, fe26b209fc9842d28d31c6243c869061; no voters:
I20240502 14:14:50.304533 23354 raft_consensus.cc:2781] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.304653 23354 raft_consensus.cc:482] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.304791 23354 raft_consensus.cc:3037] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.307646 23354 raft_consensus.cc:504] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.308238 23354 leader_election.cc:288] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Requested vote from peers fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657), 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981)
I20240502 14:14:50.308780 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d265043b247b4a708a68ade84ef78" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26b209fc9842d28d31c6243c869061"
I20240502 14:14:50.308908 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d265043b247b4a708a68ade84ef78" candidate_uuid: "491bfc422e6440749f54ffffdb717e3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597f2b4f10754298a4a428ccd386c7d3"
I20240502 14:14:50.309021 22053 raft_consensus.cc:3037] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.309103 22314 raft_consensus.cc:3037] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.311128 22314 raft_consensus.cc:2454] T 1e7d265043b247b4a708a68ade84ef78 P 597f2b4f10754298a4a428ccd386c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:50.311957 22053 raft_consensus.cc:2454] T 1e7d265043b247b4a708a68ade84ef78 P fe26b209fc9842d28d31c6243c869061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491bfc422e6440749f54ffffdb717e3a in term 1.
I20240502 14:14:50.311923 22513 leader_election.cc:302] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491bfc422e6440749f54ffffdb717e3a, 597f2b4f10754298a4a428ccd386c7d3; no voters:
I20240502 14:14:50.312253 23354 raft_consensus.cc:2781] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.312427 23354 raft_consensus.cc:686] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [term 1 LEADER]: Becoming Leader. State: Replica: 491bfc422e6440749f54ffffdb717e3a, State: Running, Role: LEADER
I20240502 14:14:50.312953 23354 consensus_queue.cc:237] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } }
I20240502 14:14:50.316025 21777 catalog_manager.cc:5484] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 491bfc422e6440749f54ffffdb717e3a (127.21.58.70). New cstate: current_term: 1 leader_uuid: "491bfc422e6440749f54ffffdb717e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:50.329583 23639 raft_consensus.cc:482] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.329768 23639 raft_consensus.cc:504] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.330402 23639 leader_election.cc:288] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:50.330803 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3906b7604e4e109010948e9c45a947" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673" is_pre_election: true
I20240502 14:14:50.330946 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3906b7604e4e109010948e9c45a947" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" is_pre_election: true
I20240502 14:14:50.331025 22446 raft_consensus.cc:2454] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:50.331187 22184 raft_consensus.cc:2454] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 0.
I20240502 14:14:50.331490 21855 leader_election.cc:302] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, fe26ecc72bed46179803517c549c5673; no voters:
I20240502 14:14:50.331766 23639 raft_consensus.cc:2781] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.331873 23639 raft_consensus.cc:482] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.331971 23639 raft_consensus.cc:3037] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.333968 23639 raft_consensus.cc:504] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.334481 23639 leader_election.cc:288] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Requested vote from peers fe26ecc72bed46179803517c549c5673 (127.21.58.69:42619), 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989)
I20240502 14:14:50.334957 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3906b7604e4e109010948e9c45a947" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb94cf6e2145c6bc6033236a27c4b8"
I20240502 14:14:50.335180 22184 raft_consensus.cc:3037] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.335181 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3906b7604e4e109010948e9c45a947" candidate_uuid: "4b0a27c175a3442dac0cae3f5b505110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26ecc72bed46179803517c549c5673"
I20240502 14:14:50.335352 22446 raft_consensus.cc:3037] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.337028 22184 raft_consensus.cc:2454] T dc3906b7604e4e109010948e9c45a947 P 6adb94cf6e2145c6bc6033236a27c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:50.337276 22446 raft_consensus.cc:2454] T dc3906b7604e4e109010948e9c45a947 P fe26ecc72bed46179803517c549c5673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0a27c175a3442dac0cae3f5b505110 in term 1.
I20240502 14:14:50.337385 21856 leader_election.cc:302] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0a27c175a3442dac0cae3f5b505110, 6adb94cf6e2145c6bc6033236a27c4b8; no voters:
I20240502 14:14:50.337694 23639 raft_consensus.cc:2781] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.337821 23639 raft_consensus.cc:686] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0a27c175a3442dac0cae3f5b505110, State: Running, Role: LEADER
I20240502 14:14:50.338097 23639 consensus_queue.cc:237] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } }
I20240502 14:14:50.341285 21777 catalog_manager.cc:5484] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "4b0a27c175a3442dac0cae3f5b505110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0a27c175a3442dac0cae3f5b505110" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 40171 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:50.678679 23646 consensus_queue.cc:1035] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491bfc422e6440749f54ffffdb717e3a" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.685513 23645 consensus_queue.cc:1035] T e9034fb87e834c2fa59c6d7b549b19dc P fe26b209fc9842d28d31c6243c869061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:50.723520 23685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:50.723796 23685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:50.731527 23685 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 14:14:50.861649 23639 consensus_queue.cc:1035] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26ecc72bed46179803517c549c5673" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.896595 23475 consensus_queue.cc:1035] T dc3906b7604e4e109010948e9c45a947 P 4b0a27c175a3442dac0cae3f5b505110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb94cf6e2145c6bc6033236a27c4b8" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.935424 23682 consensus_queue.cc:1035] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "597f2b4f10754298a4a428ccd386c7d3" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.946426 23682 consensus_queue.cc:1035] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26b209fc9842d28d31c6243c869061" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 416.296 ms
time per row: 0.208148 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:52.763908 22557 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:52.764608 21901 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:52.779701 22163 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:52.798105 22294 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:52.811617 22032 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:52.844599 22426 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:53.071862 21901 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.074944 22557 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:53.094347 22294 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:53.114960 22163 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.122479 22032 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.122896 22426 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:53.372809 22557 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:53.383932 22294 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:53.394909 22032 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.411509 21901 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.418515 22426 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:53.450800 22163 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.664253 22557 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:53.670043 22032 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.673681 22294 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:53.675573 22163 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.675989 21901 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.705775 22426 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:53.958268 21901 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.964603 22294 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:53.984618 22163 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:53.997965 22426 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:54.002581 22032 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:54.004626 22557 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:54.417382 22294 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:54.422678 22557 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:54.424499 21901 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:54.441885 22163 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:54.452245 22426 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:54.477494 22032 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
89f88a26537f4da7a7cf931ff11e658c | 127.21.58.126:41863 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:43119 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
491bfc422e6440749f54ffffdb717e3a | 127.21.58.70:40085 | HEALTHY | /C | 10 | 0
4b0a27c175a3442dac0cae3f5b505110 | 127.21.58.65:40171 | HEALTHY | /A | 6 | 0
597f2b4f10754298a4a428ccd386c7d3 | 127.21.58.68:33981 | HEALTHY | /B | 3 | 0
6adb94cf6e2145c6bc6033236a27c4b8 | 127.21.58.67:43989 | HEALTHY | /B | 6 | 0
fe26b209fc9842d28d31c6243c869061 | 127.21.58.66:40657 | HEALTHY | /A | 6 | 0
fe26ecc72bed46179803517c549c5673 | 127.21.58.69:42619 | HEALTHY | /C | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/B | 2
/C | 2
/A | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.21.58.65 | experimental | 127.21.58.65:40171
local_ip_for_outbound_sockets | 127.21.58.66 | experimental | 127.21.58.66:40657
local_ip_for_outbound_sockets | 127.21.58.67 | experimental | 127.21.58.67:43989
local_ip_for_outbound_sockets | 127.21.58.68 | experimental | 127.21.58.68:33981
local_ip_for_outbound_sockets | 127.21.58.69 | experimental | 127.21.58.69:42619
local_ip_for_outbound_sockets | 127.21.58.70 | experimental | 127.21.58.70:40085
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.58.65:40171
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.58.66:40657
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.58.67:43989
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.21.58.68:33981
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.21.58.69:42619
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.21.58.70:40085
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:43119 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:14:54.544052 21737 log_verifier.cc:126] Checking tablet 15f020ceb22f488991c3f0b36e3d9bf2
I20240502 14:14:54.588471 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 15f020ceb22f488991c3f0b36e3d9bf2
I20240502 14:14:54.588654 21737 log_verifier.cc:126] Checking tablet 161ee37964b64237b85db9e5f298a9d8
I20240502 14:14:54.627272 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 161ee37964b64237b85db9e5f298a9d8
I20240502 14:14:54.627408 21737 log_verifier.cc:126] Checking tablet 1df7a948dbd24e29812175b51c325ca2
I20240502 14:14:54.668308 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1df7a948dbd24e29812175b51c325ca2
I20240502 14:14:54.668427 21737 log_verifier.cc:126] Checking tablet 1e7d265043b247b4a708a68ade84ef78
I20240502 14:14:54.707795 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1e7d265043b247b4a708a68ade84ef78
I20240502 14:14:54.707952 21737 log_verifier.cc:126] Checking tablet 20e53ba0076940e0942889748251188a
I20240502 14:14:54.755167 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 20e53ba0076940e0942889748251188a
I20240502 14:14:54.755281 21737 log_verifier.cc:126] Checking tablet 2727235f406a456cbff925bda1f1cddb
I20240502 14:14:54.800482 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2727235f406a456cbff925bda1f1cddb
I20240502 14:14:54.800592 21737 log_verifier.cc:126] Checking tablet 27d0e5f7218146ba898af46e0caea1e4
I20240502 14:14:54.845715 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 27d0e5f7218146ba898af46e0caea1e4
I20240502 14:14:54.845840 21737 log_verifier.cc:126] Checking tablet 2e96ad6ded944064891578165429cc82
I20240502 14:14:54.892431 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2e96ad6ded944064891578165429cc82
I20240502 14:14:54.892553 21737 log_verifier.cc:126] Checking tablet 331bc60cc22b4081b23e763e45bf581c
I20240502 14:14:55.012984 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 331bc60cc22b4081b23e763e45bf581c
I20240502 14:14:55.013108 21737 log_verifier.cc:126] Checking tablet 387d9f2366444337a89005fccbc87128
I20240502 14:14:55.061866 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 387d9f2366444337a89005fccbc87128
I20240502 14:14:55.061983 21737 log_verifier.cc:126] Checking tablet 38de82b0fbb54b8280debbea89a57432
I20240502 14:14:55.109066 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 38de82b0fbb54b8280debbea89a57432
I20240502 14:14:55.109272 21737 log_verifier.cc:126] Checking tablet 3f92441b03ba422e9228060703d78625
I20240502 14:14:55.153399 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3f92441b03ba422e9228060703d78625
I20240502 14:14:55.153529 21737 log_verifier.cc:126] Checking tablet 40ec36463b2e43b887a2da7747de152e
I20240502 14:14:55.199497 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 40ec36463b2e43b887a2da7747de152e
I20240502 14:14:55.199630 21737 log_verifier.cc:126] Checking tablet 44890bee90ca4ea0aa16acbd2f3dfcf2
I20240502 14:14:55.247884 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 44890bee90ca4ea0aa16acbd2f3dfcf2
I20240502 14:14:55.248005 21737 log_verifier.cc:126] Checking tablet 4d58f9f4a8b44df481f4093bcb7fe2d7
I20240502 14:14:55.297067 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4d58f9f4a8b44df481f4093bcb7fe2d7
I20240502 14:14:55.297188 21737 log_verifier.cc:126] Checking tablet 5492c913c51d43afbf165205127a172e
I20240502 14:14:55.343433 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5492c913c51d43afbf165205127a172e
I20240502 14:14:55.343537 21737 log_verifier.cc:126] Checking tablet 572d3a690c944995a666e919b92d1814
I20240502 14:14:55.388561 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 572d3a690c944995a666e919b92d1814
I20240502 14:14:55.388689 21737 log_verifier.cc:126] Checking tablet 6820fc7861c9495388ac1458eb4956c6
I20240502 14:14:55.433151 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6820fc7861c9495388ac1458eb4956c6
I20240502 14:14:55.433275 21737 log_verifier.cc:126] Checking tablet 699a11fb57824dfbbc243eee82a68777
I20240502 14:14:55.478328 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 699a11fb57824dfbbc243eee82a68777
I20240502 14:14:55.478456 21737 log_verifier.cc:126] Checking tablet 81c891f8b1e54990b803493f459151ca
I20240502 14:14:55.525619 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 81c891f8b1e54990b803493f459151ca
I20240502 14:14:55.525727 21737 log_verifier.cc:126] Checking tablet 84d7d6aac46946bd8c27cc663a0bfa0b
I20240502 14:14:55.568187 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 84d7d6aac46946bd8c27cc663a0bfa0b
I20240502 14:14:55.568291 21737 log_verifier.cc:126] Checking tablet 925a4efd9c1040a4a1303ee999a5c995
I20240502 14:14:55.607210 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 925a4efd9c1040a4a1303ee999a5c995
I20240502 14:14:55.607316 21737 log_verifier.cc:126] Checking tablet 9ae862f0ebab4b078f60aae9c4379b5d
I20240502 14:14:55.646755 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9ae862f0ebab4b078f60aae9c4379b5d
I20240502 14:14:55.646868 21737 log_verifier.cc:126] Checking tablet a188e47225bc4fafa214540472d88d26
I20240502 14:14:55.693233 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a188e47225bc4fafa214540472d88d26
I20240502 14:14:55.693346 21737 log_verifier.cc:126] Checking tablet a33dd28a2c974ce4b16ae21107ccdc7f
I20240502 14:14:55.739054 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a33dd28a2c974ce4b16ae21107ccdc7f
I20240502 14:14:55.739204 21737 log_verifier.cc:126] Checking tablet a51eb2447a234f44b89daf1af6d1ea87
I20240502 14:14:55.785531 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a51eb2447a234f44b89daf1af6d1ea87
I20240502 14:14:55.785648 21737 log_verifier.cc:126] Checking tablet a688ca99359c4f1ead97f159aa59e14c
I20240502 14:14:55.829807 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a688ca99359c4f1ead97f159aa59e14c
I20240502 14:14:55.829917 21737 log_verifier.cc:126] Checking tablet b5c3ece6f2614827b2b5abd95228ac6d
I20240502 14:14:55.875875 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b5c3ece6f2614827b2b5abd95228ac6d
I20240502 14:14:55.875989 21737 log_verifier.cc:126] Checking tablet b8476984ef484ea9900f117ca2aed7de
I20240502 14:14:55.923238 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b8476984ef484ea9900f117ca2aed7de
I20240502 14:14:55.923348 21737 log_verifier.cc:126] Checking tablet bdecd74592ee4fa695f8d458954e1fcd
I20240502 14:14:55.966651 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bdecd74592ee4fa695f8d458954e1fcd
I20240502 14:14:55.966763 21737 log_verifier.cc:126] Checking tablet c8ecbdc3e6bd440dabe6a4107c8599ca
I20240502 14:14:56.011485 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c8ecbdc3e6bd440dabe6a4107c8599ca
I20240502 14:14:56.011615 21737 log_verifier.cc:126] Checking tablet dc3906b7604e4e109010948e9c45a947
I20240502 14:14:56.055768 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dc3906b7604e4e109010948e9c45a947
I20240502 14:14:56.055902 21737 log_verifier.cc:126] Checking tablet dce65a3c4f1744a38cadc96f79c0c084
I20240502 14:14:56.099807 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dce65a3c4f1744a38cadc96f79c0c084
I20240502 14:14:56.099982 21737 log_verifier.cc:126] Checking tablet e9034fb87e834c2fa59c6d7b549b19dc
I20240502 14:14:56.141788 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e9034fb87e834c2fa59c6d7b549b19dc
I20240502 14:14:56.141898 21737 log_verifier.cc:126] Checking tablet e999c7d734334277b470ac0ebbee304f
I20240502 14:14:56.185981 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e999c7d734334277b470ac0ebbee304f
I20240502 14:14:56.186091 21737 log_verifier.cc:126] Checking tablet e9b5501053f14bd29349ef1e3af43ec0
I20240502 14:14:56.230307 21737 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e9b5501053f14bd29349ef1e3af43ec0
I20240502 14:14:56.250862 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 21839
I20240502 14:14:56.257280 21961 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:56.392887 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 21839
W20240502 14:14:56.416989 22379 connection.cc:535] client connection to 127.21.58.65:40171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:14:56.417268 22379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:14:56.417392 22117 connection.cc:535] client connection to 127.21.58.65:40171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:14:56.417605 22117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:14:56.417681 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 21970
W20240502 14:14:56.423839 22119 connection.cc:535] server connection from 127.21.58.65:41979 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:14:56.426110 22117 consensus_peers.cc:485] T 331bc60cc22b4081b23e763e45bf581c P 6adb94cf6e2145c6bc6033236a27c4b8 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.426975 22379 consensus_peers.cc:485] T dce65a3c4f1744a38cadc96f79c0c084 P fe26ecc72bed46179803517c549c5673 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.427510 22117 consensus_peers.cc:485] T 81c891f8b1e54990b803493f459151ca P 6adb94cf6e2145c6bc6033236a27c4b8 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.430958 22093 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:14:56.479001 22117 consensus_peers.cc:485] T b5c3ece6f2614827b2b5abd95228ac6d P 6adb94cf6e2145c6bc6033236a27c4b8 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.497402 22379 consensus_peers.cc:485] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.511453 22379 consensus_peers.cc:485] T 1df7a948dbd24e29812175b51c325ca2 P fe26ecc72bed46179803517c549c5673 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.566030 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 21970
W20240502 14:14:56.580649 22117 consensus_peers.cc:485] T 161ee37964b64237b85db9e5f298a9d8 P 6adb94cf6e2145c6bc6033236a27c4b8 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.591287 22511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.58.66:40657 (error 108)
I20240502 14:14:56.591492 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 22101
W20240502 14:14:56.591481 22248 connection.cc:535] server connection from 127.21.58.66:54535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:14:56.593317 22379 consensus_peers.cc:485] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.593868 22511 consensus_peers.cc:485] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.594013 22511 consensus_peers.cc:485] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.598032 22223 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:14:56.612704 22249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111)
W20240502 14:14:56.613922 22249 consensus_peers.cc:485] T 3f92441b03ba422e9228060703d78625 P 597f2b4f10754298a4a428ccd386c7d3 -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.619153 22249 consensus_peers.cc:485] T 572d3a690c944995a666e919b92d1814 P 597f2b4f10754298a4a428ccd386c7d3 -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.633797 22249 consensus_peers.cc:485] T 5492c913c51d43afbf165205127a172e P 597f2b4f10754298a4a428ccd386c7d3 -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.696343 22510 consensus_peers.cc:485] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.716233 22511 consensus_peers.cc:485] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.732030 22511 consensus_peers.cc:485] T c8ecbdc3e6bd440dabe6a4107c8599ca P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.736598 22511 consensus_peers.cc:485] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.741277 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 22101
W20240502 14:14:56.748116 22511 consensus_peers.cc:485] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.767524 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 22232
W20240502 14:14:56.767940 22381 consensus_peers.cc:485] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 -> Peer 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989): Couldn't send request to peer 6adb94cf6e2145c6bc6033236a27c4b8. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.67:43989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.773883 22354 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:14:56.812057 22511 consensus_peers.cc:485] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.842062 22379 consensus_peers.cc:485] T 44890bee90ca4ea0aa16acbd2f3dfcf2 P fe26ecc72bed46179803517c549c5673 -> Peer 4b0a27c175a3442dac0cae3f5b505110 (127.21.58.65:40171): Couldn't send request to peer 4b0a27c175a3442dac0cae3f5b505110. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.859865 22381 consensus_peers.cc:485] T 2727235f406a456cbff925bda1f1cddb P fe26ecc72bed46179803517c549c5673 -> Peer 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989): Couldn't send request to peer 6adb94cf6e2145c6bc6033236a27c4b8. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.67:43989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.895604 22381 consensus_peers.cc:485] T e9b5501053f14bd29349ef1e3af43ec0 P fe26ecc72bed46179803517c549c5673 -> Peer 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989): Couldn't send request to peer 6adb94cf6e2145c6bc6033236a27c4b8. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.67:43989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.907744 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 22232
W20240502 14:14:56.912070 22511 consensus_peers.cc:485] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.916476 22512 consensus_peers.cc:485] T e999c7d734334277b470ac0ebbee304f P 491bfc422e6440749f54ffffdb717e3a -> Peer 6adb94cf6e2145c6bc6033236a27c4b8 (127.21.58.67:43989): Couldn't send request to peer 6adb94cf6e2145c6bc6033236a27c4b8. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.67:43989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.920598 22511 consensus_peers.cc:485] T 20e53ba0076940e0942889748251188a P 491bfc422e6440749f54ffffdb717e3a -> Peer fe26b209fc9842d28d31c6243c869061 (127.21.58.66:40657): Couldn't send request to peer fe26b209fc9842d28d31c6243c869061. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:40657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.933636 22513 connection.cc:535] client connection to 127.21.58.68:33981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:14:56.934053 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 22363
W20240502 14:14:56.934670 22513 consensus_peers.cc:485] T a51eb2447a234f44b89daf1af6d1ea87 P 491bfc422e6440749f54ffffdb717e3a -> Peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Couldn't send request to peer 597f2b4f10754298a4a428ccd386c7d3. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:33981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.934847 22513 consensus_peers.cc:485] T a188e47225bc4fafa214540472d88d26 P 491bfc422e6440749f54ffffdb717e3a -> Peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Couldn't send request to peer 597f2b4f10754298a4a428ccd386c7d3. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:33981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:56.934945 22513 consensus_peers.cc:485] T 1e7d265043b247b4a708a68ade84ef78 P 491bfc422e6440749f54ffffdb717e3a -> Peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Couldn't send request to peer 597f2b4f10754298a4a428ccd386c7d3. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:33981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.940475 22485 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:14:56.983498 22513 consensus_peers.cc:485] T 2e96ad6ded944064891578165429cc82 P 491bfc422e6440749f54ffffdb717e3a -> Peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Couldn't send request to peer 597f2b4f10754298a4a428ccd386c7d3. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:33981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:57.031841 22513 consensus_peers.cc:485] T 699a11fb57824dfbbc243eee82a68777 P 491bfc422e6440749f54ffffdb717e3a -> Peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Couldn't send request to peer 597f2b4f10754298a4a428ccd386c7d3. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:33981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:57.074980 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 22363
W20240502 14:14:57.101233 22513 consensus_peers.cc:485] T b8476984ef484ea9900f117ca2aed7de P 491bfc422e6440749f54ffffdb717e3a -> Peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Couldn't send request to peer 597f2b4f10754298a4a428ccd386c7d3. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:33981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:57.107558 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 22494
W20240502 14:14:57.107751 22513 consensus_peers.cc:485] T 15f020ceb22f488991c3f0b36e3d9bf2 P 491bfc422e6440749f54ffffdb717e3a -> Peer 597f2b4f10754298a4a428ccd386c7d3 (127.21.58.68:33981): Couldn't send request to peer 597f2b4f10754298a4a428ccd386c7d3. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:33981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:57.116974 22616 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:57.253470 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 22494
W20240502 14:14:57.278625 21764 connection.cc:535] server connection from 127.21.58.70:42279 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:14:57.278904 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 21749
I20240502 14:14:57.279887 21810 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:57.432965 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 21749
2024-05-02T14:14:57Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (28946 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (28946 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240502 14:14:57.590564 21737 test_util.cc:268] Using random seed: 978844929
I20240502 14:14:57.592226 21737 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:14:57.592289 21737 ts_itest-base.cc:113] --------------
I20240502 14:14:57.592332 21737 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:14:57.592389 21737 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:14:57.592428 21737 ts_itest-base.cc:116] --------------
2024-05-02T14:14:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:14:57Z Disabled control of system clock
I20240502 14:14:57.627203 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:35293
--webserver_interface=127.21.58.126
--webserver_port=0
--builtin_ntp_servers=127.21.58.84:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.58.126:35293
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:14:57.953411 24089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:57.953693 24089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:57.953781 24089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:57.960738 24089 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:57.960834 24089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:14:57.960873 24089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:57.960919 24089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:14:57.960963 24089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:14:57.969347 24089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.58.126:35293
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.58.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:57.970469 24089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:57.972121 24089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:57.982496 24094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:57.983187 24095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:57.983711 24097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:57.984428 24089 server_base.cc:1034] running on GCE node
I20240502 14:14:57.985103 24089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:57.986384 24089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:57.987587 24089 hybrid_clock.cc:648] HybridClock initialized: now 1714659297987543 us; error 31 us; skew 500 ppm
I20240502 14:14:57.987915 24089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:57.989925 24089 webserver.cc:460] Webserver started at http://127.21.58.126:38785/ using document root <none> and password file <none>
I20240502 14:14:57.990577 24089 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:57.990698 24089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:57.990998 24089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:57.993304 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3"
format_stamp: "Formatted at 2024-05-02 14:14:57 on dist-test-slave-wtgm"
I20240502 14:14:57.993995 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3"
format_stamp: "Formatted at 2024-05-02 14:14:57 on dist-test-slave-wtgm"
I20240502 14:14:57.998767 24089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 14:14:58.001801 24103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:58.003000 24089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:58.003193 24089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3"
format_stamp: "Formatted at 2024-05-02 14:14:57 on dist-test-slave-wtgm"
I20240502 14:14:58.003394 24089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:58.019634 24089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:58.020310 24089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:58.020562 24089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:58.037259 24154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.126:35293 every 8 connection(s)
I20240502 14:14:58.037241 24089 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.126:35293
I20240502 14:14:58.038470 24089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:14:58.041846 24155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.044543 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 24089
I20240502 14:14:58.044751 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:14:58.051497 24155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3: Bootstrap starting.
I20240502 14:14:58.054772 24155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.055886 24155 log.cc:824] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:58.058225 24155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3: No bootstrap required, opened a new log
I20240502 14:14:58.063832 24155 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 35293 } }
I20240502 14:14:58.064126 24155 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.064229 24155 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfa8fdb48e6495a83c0748f92ef8cd3, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.064821 24155 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 35293 } }
I20240502 14:14:58.065024 24155 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:58.065176 24155 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:58.065311 24155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.067217 24155 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 35293 } }
I20240502 14:14:58.067687 24155 leader_election.cc:302] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccfa8fdb48e6495a83c0748f92ef8cd3; no voters:
I20240502 14:14:58.068177 24155 leader_election.cc:288] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:58.068295 24160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:58.069415 24160 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [term 1 LEADER]: Becoming Leader. State: Replica: ccfa8fdb48e6495a83c0748f92ef8cd3, State: Running, Role: LEADER
I20240502 14:14:58.069841 24160 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 35293 } }
I20240502 14:14:58.069977 24155 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:58.071995 24162 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccfa8fdb48e6495a83c0748f92ef8cd3. Latest consensus state: current_term: 1 leader_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 35293 } } }
I20240502 14:14:58.072242 24162 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:58.072748 24161 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfa8fdb48e6495a83c0748f92ef8cd3" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 35293 } } }
I20240502 14:14:58.073032 24166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:58.073009 24161 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:58.086485 24166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:58.093586 24166 catalog_manager.cc:1347] Generated new cluster ID: 6c7273d71bb7462ca718a7c347f5a39a
I20240502 14:14:58.093684 24166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:58.112401 24166 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:58.113395 24166 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:58.122900 24166 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ccfa8fdb48e6495a83c0748f92ef8cd3: Generated new TSK 0
I20240502 14:14:58.123574 24166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:58.129034 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.65:0
--local_ip_for_outbound_sockets=127.21.58.65
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--builtin_ntp_servers=127.21.58.84:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:14:58.448467 24180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:58.448748 24180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:58.448885 24180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:58.456027 24180 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:58.456159 24180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:58.456308 24180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.65
I20240502 14:14:58.464756 24180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.65
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:58.465881 24180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:58.467494 24180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:58.478932 24188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:58.479598 24185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:58.480265 24186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:58.480911 24180 server_base.cc:1034] running on GCE node
I20240502 14:14:58.481436 24180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:58.482234 24180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:58.483428 24180 hybrid_clock.cc:648] HybridClock initialized: now 1714659298483376 us; error 49 us; skew 500 ppm
I20240502 14:14:58.483711 24180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:58.485801 24180 webserver.cc:460] Webserver started at http://127.21.58.65:33883/ using document root <none> and password file <none>
I20240502 14:14:58.486441 24180 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:58.486542 24180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:58.486841 24180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:58.489213 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fc8594b13007487b863654a94e16de19"
format_stamp: "Formatted at 2024-05-02 14:14:58 on dist-test-slave-wtgm"
I20240502 14:14:58.489914 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fc8594b13007487b863654a94e16de19"
format_stamp: "Formatted at 2024-05-02 14:14:58 on dist-test-slave-wtgm"
I20240502 14:14:58.494509 24180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 14:14:58.497499 24194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:58.498613 24180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:58.498766 24180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fc8594b13007487b863654a94e16de19"
format_stamp: "Formatted at 2024-05-02 14:14:58 on dist-test-slave-wtgm"
I20240502 14:14:58.498934 24180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:58.530412 24180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:58.531067 24180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:58.531294 24180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:58.532127 24180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:58.533445 24180 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:58.533527 24180 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:58.533603 24180 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:58.533655 24180 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:58.563853 24180 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.65:33963
I20240502 14:14:58.563921 24306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.65:33963 every 8 connection(s)
I20240502 14:14:58.565006 24180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:14:58.573168 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 24180
I20240502 14:14:58.573340 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:14:58.576603 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.66:0
--local_ip_for_outbound_sockets=127.21.58.66
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--builtin_ntp_servers=127.21.58.84:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:58.584183 24307 heartbeater.cc:347] Connected to a master server at 127.21.58.126:35293
I20240502 14:14:58.584486 24307 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:58.585352 24307 heartbeater.cc:511] Master 127.21.58.126:35293 requested a full tablet report, sending...
I20240502 14:14:58.587563 24120 ts_manager.cc:194] Registered new tserver with Master: fc8594b13007487b863654a94e16de19 (127.21.58.65:33963)
I20240502 14:14:58.588985 24120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.65:48895
W20240502 14:14:58.885419 24311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:58.885660 24311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:58.885792 24311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:58.893129 24311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:58.893225 24311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:58.893375 24311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.66
I20240502 14:14:58.901757 24311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.66
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:58.902980 24311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:58.904935 24311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:58.918125 24316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:58.918885 24319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:58.919559 24317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:58.920225 24311 server_base.cc:1034] running on GCE node
I20240502 14:14:58.920742 24311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:58.921558 24311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:58.922755 24311 hybrid_clock.cc:648] HybridClock initialized: now 1714659298922703 us; error 42 us; skew 500 ppm
I20240502 14:14:58.923018 24311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:58.924937 24311 webserver.cc:460] Webserver started at http://127.21.58.66:43357/ using document root <none> and password file <none>
I20240502 14:14:58.925530 24311 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:58.925652 24311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:58.925951 24311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:58.928349 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2a591411457045549e28da6ffa8fcb23"
format_stamp: "Formatted at 2024-05-02 14:14:58 on dist-test-slave-wtgm"
I20240502 14:14:58.929090 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2a591411457045549e28da6ffa8fcb23"
format_stamp: "Formatted at 2024-05-02 14:14:58 on dist-test-slave-wtgm"
I20240502 14:14:58.933692 24311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:14:58.936533 24325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:58.937659 24311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:58.937834 24311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2a591411457045549e28da6ffa8fcb23"
format_stamp: "Formatted at 2024-05-02 14:14:58 on dist-test-slave-wtgm"
I20240502 14:14:58.938007 24311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:58.959750 24311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:58.960414 24311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:58.960656 24311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:58.961452 24311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:58.962747 24311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:58.962839 24311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:58.962946 24311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:58.963021 24311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:58.992395 24311 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.66:37629
I20240502 14:14:58.992448 24437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.66:37629 every 8 connection(s)
I20240502 14:14:58.993480 24311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:14:58.999392 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 24311
I20240502 14:14:58.999562 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:14:59.002642 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:0
--local_ip_for_outbound_sockets=127.21.58.67
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--builtin_ntp_servers=127.21.58.84:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:59.005057 24438 heartbeater.cc:347] Connected to a master server at 127.21.58.126:35293
I20240502 14:14:59.005298 24438 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:59.005997 24438 heartbeater.cc:511] Master 127.21.58.126:35293 requested a full tablet report, sending...
I20240502 14:14:59.007288 24120 ts_manager.cc:194] Registered new tserver with Master: 2a591411457045549e28da6ffa8fcb23 (127.21.58.66:37629)
I20240502 14:14:59.008096 24120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.66:35455
W20240502 14:14:59.321866 24441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:59.322090 24441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:59.322168 24441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:59.329135 24441 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:59.329245 24441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:59.329358 24441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:14:59.337584 24441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:59.338696 24441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:59.340286 24441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:59.351692 24446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:59.352288 24447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:59.352826 24449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:59.353461 24441 server_base.cc:1034] running on GCE node
I20240502 14:14:59.353919 24441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:59.354676 24441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:59.355865 24441 hybrid_clock.cc:648] HybridClock initialized: now 1714659299355817 us; error 43 us; skew 500 ppm
I20240502 14:14:59.356132 24441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:59.358049 24441 webserver.cc:460] Webserver started at http://127.21.58.67:33321/ using document root <none> and password file <none>
I20240502 14:14:59.358695 24441 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:59.358824 24441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:59.359145 24441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:59.361397 24441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8e95044965d743328de87139aa221f0f"
format_stamp: "Formatted at 2024-05-02 14:14:59 on dist-test-slave-wtgm"
I20240502 14:14:59.362066 24441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8e95044965d743328de87139aa221f0f"
format_stamp: "Formatted at 2024-05-02 14:14:59 on dist-test-slave-wtgm"
I20240502 14:14:59.366865 24441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 14:14:59.369988 24455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:59.371163 24441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:14:59.371302 24441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8e95044965d743328de87139aa221f0f"
format_stamp: "Formatted at 2024-05-02 14:14:59 on dist-test-slave-wtgm"
I20240502 14:14:59.371470 24441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:59.382566 24441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:59.383313 24441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:59.383550 24441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:59.384420 24441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:59.385715 24441 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:59.385814 24441 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:59.385917 24441 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:59.385996 24441 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:59.415962 24441 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:44673
I20240502 14:14:59.416016 24567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:44673 every 8 connection(s)
I20240502 14:14:59.417119 24441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:14:59.424609 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 24441
I20240502 14:14:59.424769 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:14:59.427917 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:0
--local_ip_for_outbound_sockets=127.21.58.68
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--builtin_ntp_servers=127.21.58.84:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:59.428714 24568 heartbeater.cc:347] Connected to a master server at 127.21.58.126:35293
I20240502 14:14:59.429018 24568 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:59.429963 24568 heartbeater.cc:511] Master 127.21.58.126:35293 requested a full tablet report, sending...
I20240502 14:14:59.431380 24120 ts_manager.cc:194] Registered new tserver with Master: 8e95044965d743328de87139aa221f0f (127.21.58.67:44673)
I20240502 14:14:59.432127 24120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:56445
I20240502 14:14:59.591818 24307 heartbeater.cc:503] Master 127.21.58.126:35293 was elected leader, sending a full tablet report...
W20240502 14:14:59.734983 24571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:59.735280 24571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:59.735409 24571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:59.742115 24571 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:59.742224 24571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:59.742353 24571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:14:59.750698 24571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:59.751958 24571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:59.753449 24571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:59.763954 24576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:59.764534 24577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:59.765012 24579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:59.765632 24571 server_base.cc:1034] running on GCE node
I20240502 14:14:59.766156 24571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:59.766940 24571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:59.768121 24571 hybrid_clock.cc:648] HybridClock initialized: now 1714659299768078 us; error 33 us; skew 500 ppm
I20240502 14:14:59.768388 24571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:59.770457 24571 webserver.cc:460] Webserver started at http://127.21.58.68:45277/ using document root <none> and password file <none>
I20240502 14:14:59.771180 24571 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:59.771281 24571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:59.771579 24571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:59.774004 24571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "829ec7d7d701401594888bbe75c3dbbc"
format_stamp: "Formatted at 2024-05-02 14:14:59 on dist-test-slave-wtgm"
I20240502 14:14:59.774729 24571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "829ec7d7d701401594888bbe75c3dbbc"
format_stamp: "Formatted at 2024-05-02 14:14:59 on dist-test-slave-wtgm"
I20240502 14:14:59.779693 24571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 14:14:59.782629 24585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:59.783903 24571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:59.784061 24571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "829ec7d7d701401594888bbe75c3dbbc"
format_stamp: "Formatted at 2024-05-02 14:14:59 on dist-test-slave-wtgm"
I20240502 14:14:59.784229 24571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:59.806398 24571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:59.807147 24571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:59.807365 24571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:59.808245 24571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:59.809502 24571 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:59.809579 24571 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:59.809676 24571 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:59.809773 24571 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:59.837184 24571 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:45041
I20240502 14:14:59.837240 24697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:45041 every 8 connection(s)
I20240502 14:14:59.838347 24571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:14:59.840567 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 24571
I20240502 14:14:59.840759 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:14:59.843663 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:0
--local_ip_for_outbound_sockets=127.21.58.69
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--builtin_ntp_servers=127.21.58.84:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:59.850239 24698 heartbeater.cc:347] Connected to a master server at 127.21.58.126:35293
I20240502 14:14:59.850551 24698 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:59.851526 24698 heartbeater.cc:511] Master 127.21.58.126:35293 requested a full tablet report, sending...
I20240502 14:14:59.852917 24120 ts_manager.cc:194] Registered new tserver with Master: 829ec7d7d701401594888bbe75c3dbbc (127.21.58.68:45041)
I20240502 14:14:59.853648 24120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:38483
I20240502 14:15:00.010448 24438 heartbeater.cc:503] Master 127.21.58.126:35293 was elected leader, sending a full tablet report...
W20240502 14:15:00.162930 24701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:00.163229 24701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:00.163357 24701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:00.173631 24701 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:00.173753 24701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:00.173902 24701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:15:00.182404 24701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:35293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:00.183748 24701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:00.185413 24701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:00.196591 24706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:00.197093 24709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:00.197630 24707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:00.198247 24701 server_base.cc:1034] running on GCE node
I20240502 14:15:00.198724 24701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:00.199527 24701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:00.200724 24701 hybrid_clock.cc:648] HybridClock initialized: now 1714659300200680 us; error 37 us; skew 500 ppm
I20240502 14:15:00.201004 24701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:00.203004 24701 webserver.cc:460] Webserver started at http://127.21.58.69:36695/ using document root <none> and password file <none>
I20240502 14:15:00.203639 24701 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:00.203742 24701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:00.204016 24701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:00.206346 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8051b012edea4afb99b3af58767eaa65"
format_stamp: "Formatted at 2024-05-02 14:15:00 on dist-test-slave-wtgm"
I20240502 14:15:00.207047 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8051b012edea4afb99b3af58767eaa65"
format_stamp: "Formatted at 2024-05-02 14:15:00 on dist-test-slave-wtgm"
I20240502 14:15:00.211599 24701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 14:15:00.214402 24715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:00.215641 24701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:00.215811 24701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8051b012edea4afb99b3af58767eaa65"
format_stamp: "Formatted at 2024-05-02 14:15:00 on dist-test-slave-wtgm"
I20240502 14:15:00.215994 24701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:00.229470 24701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:00.230185 24701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:00.230420 24701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:00.231281 24701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:00.232596 24701 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:00.232697 24701 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:00.232800 24701 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:00.232870 24701 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:00.263523 24701 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:35651
I20240502 14:15:00.263579 24827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:35651 every 8 connection(s)
I20240502 14:15:00.264806 24701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:15:00.266326 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 24701
I20240502 14:15:00.266479 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:15:00.276383 24828 heartbeater.cc:347] Connected to a master server at 127.21.58.126:35293
I20240502 14:15:00.276650 24828 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:00.277478 24828 heartbeater.cc:511] Master 127.21.58.126:35293 requested a full tablet report, sending...
I20240502 14:15:00.278698 24120 ts_manager.cc:194] Registered new tserver with Master: 8051b012edea4afb99b3af58767eaa65 (127.21.58.69:35651)
I20240502 14:15:00.279458 24120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:42219
I20240502 14:15:00.281792 21737 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:15:00.297915 24120 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44932:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:15:00.327697 24503 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2733d3f5fa428cb45f1fed300ebbce (DEFAULT_TABLE table=TestTable [id=dc896f6f45df4a64a344166aad96475c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.328119 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2733d3f5fa428cb45f1fed300ebbce (DEFAULT_TABLE table=TestTable [id=dc896f6f45df4a64a344166aad96475c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.329264 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2733d3f5fa428cb45f1fed300ebbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.329272 24503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2733d3f5fa428cb45f1fed300ebbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.331990 24242 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2733d3f5fa428cb45f1fed300ebbce (DEFAULT_TABLE table=TestTable [id=dc896f6f45df4a64a344166aad96475c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.333936 24242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2733d3f5fa428cb45f1fed300ebbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.339151 24846 tablet_bootstrap.cc:492] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f: Bootstrap starting.
I20240502 14:15:00.341939 24847 tablet_bootstrap.cc:492] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc: Bootstrap starting.
I20240502 14:15:00.342595 24846 tablet_bootstrap.cc:654] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.343860 24846 log.cc:824] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:00.345616 24847 tablet_bootstrap.cc:654] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.346715 24846 tablet_bootstrap.cc:492] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f: No bootstrap required, opened a new log
I20240502 14:15:00.346789 24847 log.cc:824] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:00.346968 24846 ts_tablet_manager.cc:1367] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:15:00.347784 24849 tablet_bootstrap.cc:492] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19: Bootstrap starting.
I20240502 14:15:00.350528 24849 tablet_bootstrap.cc:654] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.351420 24849 log.cc:824] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:00.351644 24847 tablet_bootstrap.cc:492] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc: No bootstrap required, opened a new log
I20240502 14:15:00.351879 24847 ts_tablet_manager.cc:1367] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:15:00.353505 24849 tablet_bootstrap.cc:492] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19: No bootstrap required, opened a new log
I20240502 14:15:00.353729 24849 ts_tablet_manager.cc:1367] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 14:15:00.356335 24846 raft_consensus.cc:348] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.356734 24846 raft_consensus.cc:374] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.356839 24846 raft_consensus.cc:729] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e95044965d743328de87139aa221f0f, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.357471 24846 consensus_queue.cc:260] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.358495 24568 heartbeater.cc:503] Master 127.21.58.126:35293 was elected leader, sending a full tablet report...
I20240502 14:15:00.358974 24846 ts_tablet_manager.cc:1398] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20240502 14:15:00.360078 24847 raft_consensus.cc:348] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.360467 24847 raft_consensus.cc:374] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.360570 24847 raft_consensus.cc:729] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829ec7d7d701401594888bbe75c3dbbc, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.360491 24849 raft_consensus.cc:348] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.360795 24849 raft_consensus.cc:374] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.360894 24849 raft_consensus.cc:729] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8594b13007487b863654a94e16de19, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.361182 24847 consensus_queue.cc:260] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.361397 24849 consensus_queue.cc:260] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.362287 24698 heartbeater.cc:503] Master 127.21.58.126:35293 was elected leader, sending a full tablet report...
I20240502 14:15:00.362761 24847 ts_tablet_manager.cc:1398] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240502 14:15:00.364096 24849 ts_tablet_manager.cc:1398] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
W20240502 14:15:00.419198 24569 tablet.cc:2369] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:00.480648 24852 raft_consensus.cc:482] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.480877 24852 raft_consensus.cc:504] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.482443 24852 leader_election.cc:288] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8594b13007487b863654a94e16de19 (127.21.58.65:33963), 829ec7d7d701401594888bbe75c3dbbc (127.21.58.68:45041)
I20240502 14:15:00.491374 24262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2733d3f5fa428cb45f1fed300ebbce" candidate_uuid: "8e95044965d743328de87139aa221f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8594b13007487b863654a94e16de19" is_pre_election: true
I20240502 14:15:00.491375 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2733d3f5fa428cb45f1fed300ebbce" candidate_uuid: "8e95044965d743328de87139aa221f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829ec7d7d701401594888bbe75c3dbbc" is_pre_election: true
I20240502 14:15:00.491883 24262 raft_consensus.cc:2454] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e95044965d743328de87139aa221f0f in term 0.
I20240502 14:15:00.491890 24653 raft_consensus.cc:2454] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e95044965d743328de87139aa221f0f in term 0.
I20240502 14:15:00.492499 24458 leader_election.cc:302] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 829ec7d7d701401594888bbe75c3dbbc, 8e95044965d743328de87139aa221f0f; no voters:
I20240502 14:15:00.492878 24852 raft_consensus.cc:2781] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.493021 24852 raft_consensus.cc:482] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.493111 24852 raft_consensus.cc:3037] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.495014 24852 raft_consensus.cc:504] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.495605 24852 leader_election.cc:288] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [CANDIDATE]: Term 1 election: Requested vote from peers fc8594b13007487b863654a94e16de19 (127.21.58.65:33963), 829ec7d7d701401594888bbe75c3dbbc (127.21.58.68:45041)
I20240502 14:15:00.495954 24262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2733d3f5fa428cb45f1fed300ebbce" candidate_uuid: "8e95044965d743328de87139aa221f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8594b13007487b863654a94e16de19"
I20240502 14:15:00.495959 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2733d3f5fa428cb45f1fed300ebbce" candidate_uuid: "8e95044965d743328de87139aa221f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829ec7d7d701401594888bbe75c3dbbc"
I20240502 14:15:00.496137 24262 raft_consensus.cc:3037] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.496137 24653 raft_consensus.cc:3037] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.499087 24262 raft_consensus.cc:2454] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e95044965d743328de87139aa221f0f in term 1.
I20240502 14:15:00.499065 24653 raft_consensus.cc:2454] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e95044965d743328de87139aa221f0f in term 1.
I20240502 14:15:00.499459 24459 leader_election.cc:302] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e95044965d743328de87139aa221f0f, fc8594b13007487b863654a94e16de19; no voters:
I20240502 14:15:00.499706 24852 raft_consensus.cc:2781] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.500092 24852 raft_consensus.cc:686] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [term 1 LEADER]: Becoming Leader. State: Replica: 8e95044965d743328de87139aa221f0f, State: Running, Role: LEADER
I20240502 14:15:00.500488 24852 consensus_queue.cc:237] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } }
I20240502 14:15:00.504771 24120 catalog_manager.cc:5484] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e95044965d743328de87139aa221f0f (127.21.58.67). New cstate: current_term: 1 leader_uuid: "8e95044965d743328de87139aa221f0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e95044965d743328de87139aa221f0f" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:00.553385 21737 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:15:00.555421 21737 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fc8594b13007487b863654a94e16de19 to finish bootstrapping
I20240502 14:15:00.563236 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2a591411457045549e28da6ffa8fcb23 to finish bootstrapping
W20240502 14:15:00.567796 24308 tablet.cc:2369] T 9e2733d3f5fa428cb45f1fed300ebbce P fc8594b13007487b863654a94e16de19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:00.571478 21737 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8e95044965d743328de87139aa221f0f to finish bootstrapping
I20240502 14:15:00.577147 21737 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 829ec7d7d701401594888bbe75c3dbbc to finish bootstrapping
I20240502 14:15:00.583184 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8051b012edea4afb99b3af58767eaa65 to finish bootstrapping
W20240502 14:15:00.590226 24699 tablet.cc:2369] T 9e2733d3f5fa428cb45f1fed300ebbce P 829ec7d7d701401594888bbe75c3dbbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:00.900914 24852 consensus_queue.cc:1035] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "829ec7d7d701401594888bbe75c3dbbc" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.917835 24857 consensus_queue.cc:1035] T 9e2733d3f5fa428cb45f1fed300ebbce P 8e95044965d743328de87139aa221f0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8594b13007487b863654a94e16de19" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.028158 24373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.031760 24763 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.033969 24633 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.043481 24503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:01.046109 24242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.191570 24763 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.195081 24373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.201594 24633 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.206383 24503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:01.225988 24242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.281766 24828 heartbeater.cc:503] Master 127.21.58.126:35293 was elected leader, sending a full tablet report...
I20240502 14:15:01.775630 24763 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.786731 24373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.788982 24633 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.793428 24503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:01.796665 24242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.943131 24373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.947993 24763 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.959090 24503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:01.959638 24633 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:01.966711 24242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:02.131280 24763 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:02.132793 24373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:02.141930 24503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:02.147253 24633 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:02.180929 24242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:02.314823 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 24180
I20240502 14:15:02.316215 24302 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:02.408058 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 24180
I20240502 14:15:02.427196 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 24311
I20240502 14:15:02.428189 24433 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:02.514182 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 24311
I20240502 14:15:02.532512 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 24441
I20240502 14:15:02.533586 24562 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:02.625124 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 24441
W20240502 14:15:02.644001 24105 connection.cc:535] server connection from 127.21.58.67:56445 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:02.644241 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 24571
I20240502 14:15:02.645165 24693 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:02.733816 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 24571
I20240502 14:15:02.752091 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 24701
I20240502 14:15:02.753070 24823 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:02.838268 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 24701
I20240502 14:15:02.855978 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 24089
I20240502 14:15:02.857100 24150 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:02.967036 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 24089
2024-05-02T14:15:02Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5419 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5419 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240502 14:15:03.009826 21737 test_util.cc:268] Using random seed: 984264198
I20240502 14:15:03.011391 21737 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:15:03.011449 21737 ts_itest-base.cc:113] --------------
I20240502 14:15:03.011508 21737 ts_itest-base.cc:114] 7 tablet servers
I20240502 14:15:03.011556 21737 ts_itest-base.cc:115] 2 replicas per TS
I20240502 14:15:03.011620 21737 ts_itest-base.cc:116] --------------
2024-05-02T14:15:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:15:03Z Disabled control of system clock
I20240502 14:15:03.043747 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:40155
--webserver_interface=127.21.58.126
--webserver_port=0
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.58.126:40155
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 14:15:03.345294 25056 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 14:15:03.345584 25056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:03.345649 25056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:03.345746 25056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:03.352877 25056 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:03.352967 25056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:15:03.353072 25056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:03.353130 25056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:15:03.353211 25056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:15:03.361685 25056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.58.126:40155
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.58.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:03.362834 25056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:03.364375 25056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:03.374593 25061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:03.375196 25064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:03.375728 25062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:03.376381 25056 server_base.cc:1034] running on GCE node
I20240502 14:15:03.377069 25056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:03.378299 25056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:03.379498 25056 hybrid_clock.cc:648] HybridClock initialized: now 1714659303379445 us; error 27 us; skew 500 ppm
I20240502 14:15:03.379801 25056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:03.381819 25056 webserver.cc:460] Webserver started at http://127.21.58.126:41599/ using document root <none> and password file <none>
I20240502 14:15:03.382419 25056 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:03.382558 25056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:03.382823 25056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:03.385067 25056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9ab9bceb14584c9b9819e55247c54100"
format_stamp: "Formatted at 2024-05-02 14:15:03 on dist-test-slave-wtgm"
I20240502 14:15:03.385706 25056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9ab9bceb14584c9b9819e55247c54100"
format_stamp: "Formatted at 2024-05-02 14:15:03 on dist-test-slave-wtgm"
I20240502 14:15:03.390265 25056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240502 14:15:03.393118 25070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:03.394129 25056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:03.394260 25056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9ab9bceb14584c9b9819e55247c54100"
format_stamp: "Formatted at 2024-05-02 14:15:03 on dist-test-slave-wtgm"
I20240502 14:15:03.394397 25056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:03.426256 25056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:03.426923 25056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:03.427222 25056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:03.443787 25056 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.126:40155
I20240502 14:15:03.443822 25121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.126:40155 every 8 connection(s)
I20240502 14:15:03.444942 25056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:15:03.448264 25122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.452409 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25056
I20240502 14:15:03.452579 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:15:03.457803 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100: Bootstrap starting.
I20240502 14:15:03.461264 25122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.462110 25122 log.cc:824] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:03.464854 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100: No bootstrap required, opened a new log
I20240502 14:15:03.471280 25122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9bceb14584c9b9819e55247c54100" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 40155 } }
I20240502 14:15:03.471556 25122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.471644 25122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab9bceb14584c9b9819e55247c54100, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.472127 25122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9bceb14584c9b9819e55247c54100" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 40155 } }
I20240502 14:15:03.472326 25122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:15:03.472457 25122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:15:03.472626 25122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.474447 25122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9bceb14584c9b9819e55247c54100" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 40155 } }
I20240502 14:15:03.474836 25122 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab9bceb14584c9b9819e55247c54100; no voters:
I20240502 14:15:03.475344 25122 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:15:03.475440 25127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.475986 25127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [term 1 LEADER]: Becoming Leader. State: Replica: 9ab9bceb14584c9b9819e55247c54100, State: Running, Role: LEADER
I20240502 14:15:03.476723 25127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9bceb14584c9b9819e55247c54100" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 40155 } }
I20240502 14:15:03.477233 25122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:15:03.479272 25128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab9bceb14584c9b9819e55247c54100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9bceb14584c9b9819e55247c54100" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 40155 } } }
I20240502 14:15:03.479540 25128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:03.479373 25129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab9bceb14584c9b9819e55247c54100. Latest consensus state: current_term: 1 leader_uuid: "9ab9bceb14584c9b9819e55247c54100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab9bceb14584c9b9819e55247c54100" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 40155 } } }
I20240502 14:15:03.479702 25129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:03.480952 25133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:15:03.485319 25133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:15:03.493496 25133 catalog_manager.cc:1347] Generated new cluster ID: 29761fce60b748bcaeecfd2b3d10828c
I20240502 14:15:03.493588 25133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:15:03.515628 25133 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:15:03.516634 25133 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:15:03.524487 25133 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9ab9bceb14584c9b9819e55247c54100: Generated new TSK 0
I20240502 14:15:03.525132 25133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:15:03.534113 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.65:0
--local_ip_for_outbound_sockets=127.21.58.65
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:15:03.862746 25146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:03.862980 25146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:03.863060 25146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:03.869854 25146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:03.869962 25146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:03.870064 25146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.65
I20240502 14:15:03.878041 25146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.65
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:03.879112 25146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:03.880671 25146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:03.892092 25151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:03.892735 25152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:03.893249 25154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:03.893962 25146 server_base.cc:1034] running on GCE node
I20240502 14:15:03.894451 25146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:03.895256 25146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:03.896443 25146 hybrid_clock.cc:648] HybridClock initialized: now 1714659303896396 us; error 44 us; skew 500 ppm
I20240502 14:15:03.896725 25146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:03.906579 25146 webserver.cc:460] Webserver started at http://127.21.58.65:35459/ using document root <none> and password file <none>
I20240502 14:15:03.907285 25146 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:03.907410 25146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:03.907711 25146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:03.909984 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a5cb74a14a8d434e986ac7d988731e8a"
format_stamp: "Formatted at 2024-05-02 14:15:03 on dist-test-slave-wtgm"
I20240502 14:15:03.910670 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a5cb74a14a8d434e986ac7d988731e8a"
format_stamp: "Formatted at 2024-05-02 14:15:03 on dist-test-slave-wtgm"
I20240502 14:15:03.914988 25146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240502 14:15:03.918000 25160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:03.919150 25146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:03.919288 25146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a5cb74a14a8d434e986ac7d988731e8a"
format_stamp: "Formatted at 2024-05-02 14:15:03 on dist-test-slave-wtgm"
I20240502 14:15:03.919461 25146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:03.932827 25146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:03.933507 25146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:03.933773 25146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:03.934585 25146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:03.935974 25146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:03.936069 25146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:03.936177 25146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:03.936247 25146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:03.965941 25146 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.65:46045
I20240502 14:15:03.966003 25272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.65:46045 every 8 connection(s)
I20240502 14:15:03.967097 25146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:15:03.968699 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25146
I20240502 14:15:03.968916 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:15:03.972087 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.66:0
--local_ip_for_outbound_sockets=127.21.58.66
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:03.978824 25273 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:03.979151 25273 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:03.980127 25273 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:03.982113 25087 ts_manager.cc:194] Registered new tserver with Master: a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:03.983525 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.65:47119
W20240502 14:15:04.276754 25277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:04.277036 25277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:04.277170 25277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:04.283929 25277 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:04.284046 25277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:04.284185 25277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.66
I20240502 14:15:04.292485 25277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.66
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:04.293480 25277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:04.294880 25277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:04.305953 25282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.306545 25285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.307152 25283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:04.307767 25277 server_base.cc:1034] running on GCE node
I20240502 14:15:04.308269 25277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:04.309089 25277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:04.310274 25277 hybrid_clock.cc:648] HybridClock initialized: now 1714659304310238 us; error 33 us; skew 500 ppm
I20240502 14:15:04.310559 25277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:04.312574 25277 webserver.cc:460] Webserver started at http://127.21.58.66:32785/ using document root <none> and password file <none>
I20240502 14:15:04.313247 25277 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:04.313359 25277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:04.313652 25277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:04.317354 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wtgm"
I20240502 14:15:04.318084 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wtgm"
I20240502 14:15:04.322521 25277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 14:15:04.325541 25291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.326658 25277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:04.326799 25277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wtgm"
I20240502 14:15:04.326979 25277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:04.343853 25277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:04.344568 25277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:04.344818 25277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:04.345654 25277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:04.347007 25277 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:04.347134 25277 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.347255 25277 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:04.347337 25277 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.378162 25277 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.66:44323
I20240502 14:15:04.378208 25403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.66:44323 every 8 connection(s)
I20240502 14:15:04.379743 25277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:15:04.384542 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25277
I20240502 14:15:04.384725 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:15:04.388398 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:0
--local_ip_for_outbound_sockets=127.21.58.67
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:04.394201 25404 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:04.394490 25404 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:04.395450 25404 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:04.396831 25087 ts_manager.cc:194] Registered new tserver with Master: c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:04.397574 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.66:39895
W20240502 14:15:04.695693 25407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:04.695914 25407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:04.695984 25407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:04.702700 25407 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:04.702787 25407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:04.702879 25407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:15:04.710915 25407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:04.711941 25407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:04.713346 25407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:04.724084 25412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.724601 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.725157 25413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:04.725874 25407 server_base.cc:1034] running on GCE node
I20240502 14:15:04.726361 25407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:04.727183 25407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:04.728366 25407 hybrid_clock.cc:648] HybridClock initialized: now 1714659304728308 us; error 53 us; skew 500 ppm
I20240502 14:15:04.728667 25407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:04.730631 25407 webserver.cc:460] Webserver started at http://127.21.58.67:33785/ using document root <none> and password file <none>
I20240502 14:15:04.731352 25407 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:04.731460 25407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:04.731758 25407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:04.734063 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wtgm"
I20240502 14:15:04.734720 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wtgm"
I20240502 14:15:04.739233 25407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20240502 14:15:04.742096 25421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.743206 25407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:04.743340 25407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wtgm"
I20240502 14:15:04.743523 25407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:04.772607 25407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:04.773303 25407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:04.773533 25407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:04.774322 25407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:04.775547 25407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:04.775637 25407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.775734 25407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:04.775796 25407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.805148 25407 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:33255
I20240502 14:15:04.805219 25533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:33255 every 8 connection(s)
I20240502 14:15:04.806293 25407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:15:04.812422 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25407
I20240502 14:15:04.812602 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:15:04.815824 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:0
--local_ip_for_outbound_sockets=127.21.58.68
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:04.817914 25534 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:04.818145 25534 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:04.818817 25534 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:04.820214 25087 ts_manager.cc:194] Registered new tserver with Master: 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:04.821261 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:46413
I20240502 14:15:04.986443 25273 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
W20240502 14:15:05.122445 25537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:05.122697 25537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:05.122838 25537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:05.130054 25537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:05.130168 25537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:05.130273 25537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:15:05.138549 25537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:05.139616 25537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:05.141156 25537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:05.152274 25545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:05.152786 25543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:05.153311 25542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:05.153939 25537 server_base.cc:1034] running on GCE node
I20240502 14:15:05.154410 25537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:05.155212 25537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:05.156390 25537 hybrid_clock.cc:648] HybridClock initialized: now 1714659305156344 us; error 40 us; skew 500 ppm
I20240502 14:15:05.156632 25537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:05.158527 25537 webserver.cc:460] Webserver started at http://127.21.58.68:35987/ using document root <none> and password file <none>
I20240502 14:15:05.159197 25537 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:05.159287 25537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:05.159531 25537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:05.161756 25537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c69e4233b679437b8eb3b811680fa196"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:05.162415 25537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c69e4233b679437b8eb3b811680fa196"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:05.166888 25537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20240502 14:15:05.169943 25551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:05.171023 25537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:05.171173 25537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c69e4233b679437b8eb3b811680fa196"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:05.171325 25537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:05.186686 25537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:05.187392 25537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:05.187618 25537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:05.188418 25537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:05.189601 25537 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:05.189674 25537 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:05.189764 25537 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:05.189826 25537 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:05.217777 25537 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:43117
I20240502 14:15:05.217836 25663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:43117 every 8 connection(s)
I20240502 14:15:05.218814 25537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:15:05.228540 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25537
I20240502 14:15:05.228726 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:15:05.229270 25664 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:05.229502 25664 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:05.230182 25664 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:05.231420 25087 ts_manager.cc:194] Registered new tserver with Master: c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117)
I20240502 14:15:05.231514 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:0
--local_ip_for_outbound_sockets=127.21.58.69
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:05.232477 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:37089
I20240502 14:15:05.400211 25404 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
W20240502 14:15:05.539554 25667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:05.539944 25667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:05.540093 25667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:05.546572 25667 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:05.546695 25667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:05.546856 25667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:15:05.554818 25667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:05.555869 25667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:05.557518 25667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:05.568915 25673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:05.569447 25675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:05.569919 25672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:05.570501 25667 server_base.cc:1034] running on GCE node
I20240502 14:15:05.571030 25667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:05.571857 25667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:05.573061 25667 hybrid_clock.cc:648] HybridClock initialized: now 1714659305573032 us; error 27 us; skew 500 ppm
I20240502 14:15:05.573331 25667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:05.575418 25667 webserver.cc:460] Webserver started at http://127.21.58.69:43017/ using document root <none> and password file <none>
I20240502 14:15:05.576002 25667 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:05.576102 25667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:05.576395 25667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:05.578728 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:05.579413 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:05.583649 25667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:15:05.586388 25681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:05.587610 25667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:15:05.587747 25667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:05.587939 25667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:05.613871 25667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:05.614503 25667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:05.614743 25667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:05.615542 25667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:05.616823 25667 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:05.616919 25667 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:05.617031 25667 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:05.617107 25667 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:05.646802 25793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:44755 every 8 connection(s)
I20240502 14:15:05.646797 25667 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:44755
I20240502 14:15:05.648151 25667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:15:05.652383 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25667
I20240502 14:15:05.652550 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:15:05.655437 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.70:0
--local_ip_for_outbound_sockets=127.21.58.70
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:05.659978 25794 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:05.660269 25794 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:05.661186 25794 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:05.662544 25087 ts_manager.cc:194] Registered new tserver with Master: 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755)
I20240502 14:15:05.663304 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:50063
I20240502 14:15:05.823304 25534 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
W20240502 14:15:05.965287 25797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:05.965555 25797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:05.965689 25797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:05.972898 25797 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:05.973021 25797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:05.973172 25797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.70
I20240502 14:15:05.981740 25797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.70
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:05.982949 25797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:05.984591 25797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:05.995488 25802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:05.996058 25805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:05.996630 25803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:05.997228 25797 server_base.cc:1034] running on GCE node
I20240502 14:15:05.997737 25797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:05.998484 25797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:05.999670 25797 hybrid_clock.cc:648] HybridClock initialized: now 1714659305999631 us; error 43 us; skew 500 ppm
I20240502 14:15:05.999938 25797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:06.001868 25797 webserver.cc:460] Webserver started at http://127.21.58.70:40483/ using document root <none> and password file <none>
I20240502 14:15:06.002499 25797 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:06.002621 25797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:06.002907 25797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:06.005225 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6365836c0f2b428bbeb4d69b9ceda2fe"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:06.005921 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6365836c0f2b428bbeb4d69b9ceda2fe"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:06.010557 25797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20240502 14:15:06.013521 25811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:06.014613 25797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:06.014755 25797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6365836c0f2b428bbeb4d69b9ceda2fe"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:06.014930 25797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:06.026638 25797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:06.027403 25797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:06.027639 25797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:06.028421 25797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:06.029685 25797 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:06.029783 25797 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:06.029892 25797 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:06.029961 25797 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:06.059031 25797 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.70:35761
I20240502 14:15:06.059098 25923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.70:35761 every 8 connection(s)
I20240502 14:15:06.060206 25797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:15:06.067726 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25797
I20240502 14:15:06.067907 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:15:06.070850 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.71:0
--local_ip_for_outbound_sockets=127.21.58.71
--webserver_interface=127.21.58.71
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:06.071902 25924 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:06.072206 25924 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:06.073132 25924 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:06.074519 25087 ts_manager.cc:194] Registered new tserver with Master: 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70:35761)
I20240502 14:15:06.075662 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.70:52463
I20240502 14:15:06.234488 25664 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
W20240502 14:15:06.384697 25927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:06.384929 25927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:06.385011 25927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:06.391923 25927 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:06.392009 25927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:06.392109 25927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.71
I20240502 14:15:06.400048 25927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.58.71
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.71
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:06.401046 25927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:06.402673 25927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:06.413791 25933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:06.414275 25932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:06.414817 25935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:06.415354 25927 server_base.cc:1034] running on GCE node
I20240502 14:15:06.415799 25927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:06.416546 25927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:06.417737 25927 hybrid_clock.cc:648] HybridClock initialized: now 1714659306417702 us; error 33 us; skew 500 ppm
I20240502 14:15:06.418001 25927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:06.420042 25927 webserver.cc:460] Webserver started at http://127.21.58.71:39153/ using document root <none> and password file <none>
I20240502 14:15:06.420629 25927 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:06.420737 25927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:06.421000 25927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:06.423144 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "103df70e837147d9a992da0e46ece12e"
format_stamp: "Formatted at 2024-05-02 14:15:06 on dist-test-slave-wtgm"
I20240502 14:15:06.423820 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "103df70e837147d9a992da0e46ece12e"
format_stamp: "Formatted at 2024-05-02 14:15:06 on dist-test-slave-wtgm"
I20240502 14:15:06.428136 25927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:15:06.430975 25941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:06.432057 25927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:15:06.432188 25927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "103df70e837147d9a992da0e46ece12e"
format_stamp: "Formatted at 2024-05-02 14:15:06 on dist-test-slave-wtgm"
I20240502 14:15:06.432343 25927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:06.449937 25927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:06.450583 25927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:06.450822 25927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:06.451668 25927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:06.452826 25927 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:06.452911 25927 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:06.453011 25927 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:06.453080 25927 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:06.483422 25927 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.71:37231
I20240502 14:15:06.483477 26053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.71:37231 every 8 connection(s)
I20240502 14:15:06.484503 25927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:15:06.494397 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 25927
I20240502 14:15:06.494593 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:15:06.496218 26054 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:06.496522 26054 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:06.497241 26054 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:06.498507 25087 ts_manager.cc:194] Registered new tserver with Master: 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231)
I20240502 14:15:06.499322 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.71:46845
I20240502 14:15:06.509729 21737 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:15:06.523454 25087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40774:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:15:06.560070 25989 tablet_service.cc:1467] Processing CreateTablet for tablet de14694a11e14f6082bba7fa70f51553 (DEFAULT_TABLE table=TestTable [id=0445c4e8449e45c0889651a22874bc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.560818 25208 tablet_service.cc:1467] Processing CreateTablet for tablet de14694a11e14f6082bba7fa70f51553 (DEFAULT_TABLE table=TestTable [id=0445c4e8449e45c0889651a22874bc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.561251 25989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de14694a11e14f6082bba7fa70f51553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.561890 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de14694a11e14f6082bba7fa70f51553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.570746 26070 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Bootstrap starting.
I20240502 14:15:06.570966 26071 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:06.573592 26070 tablet_bootstrap.cc:654] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.573863 26071 tablet_bootstrap.cc:654] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.574909 26070 log.cc:824] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:06.574942 26071 log.cc:824] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:06.577322 26070 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: No bootstrap required, opened a new log
I20240502 14:15:06.577323 26071 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:06.577551 26071 ts_tablet_manager.cc:1367] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:15:06.577591 26070 ts_tablet_manager.cc:1367] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 14:15:06.583302 26071 raft_consensus.cc:348] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:06.583561 26071 raft_consensus.cc:374] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.583658 26071 raft_consensus.cc:729] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.583488 26070 raft_consensus.cc:348] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:06.583814 26070 raft_consensus.cc:374] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.583900 26070 raft_consensus.cc:729] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103df70e837147d9a992da0e46ece12e, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.584236 26071 consensus_queue.cc:260] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:06.584437 26070 consensus_queue.cc:260] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:06.585369 26054 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
I20240502 14:15:06.585623 26071 ts_tablet_manager.cc:1398] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 14:15:06.585888 26070 ts_tablet_manager.cc:1398] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:15:06.665581 25794 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
I20240502 14:15:06.702716 26074 raft_consensus.cc:482] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.702957 26074 raft_consensus.cc:504] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:06.704401 26074 leader_election.cc:288] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231)
I20240502 14:15:06.710737 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de14694a11e14f6082bba7fa70f51553" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103df70e837147d9a992da0e46ece12e" is_pre_election: true
I20240502 14:15:06.711194 26009 raft_consensus.cc:2454] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 0.
I20240502 14:15:06.711731 25161 leader_election.cc:302] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 103df70e837147d9a992da0e46ece12e, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:06.712082 26074 raft_consensus.cc:2781] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.712213 26074 raft_consensus.cc:482] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.712304 26074 raft_consensus.cc:3037] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.714255 26074 raft_consensus.cc:504] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:06.714659 26074 leader_election.cc:288] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Requested vote from peers 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231)
I20240502 14:15:06.715024 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de14694a11e14f6082bba7fa70f51553" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103df70e837147d9a992da0e46ece12e"
I20240502 14:15:06.715212 26009 raft_consensus.cc:3037] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.717065 26009 raft_consensus.cc:2454] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 1.
I20240502 14:15:06.717401 25161 leader_election.cc:302] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 103df70e837147d9a992da0e46ece12e, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:06.717674 26074 raft_consensus.cc:2781] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.718043 26074 raft_consensus.cc:686] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Becoming Leader. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:06.718434 26074 consensus_queue.cc:237] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
W20240502 14:15:06.720455 25274 tablet.cc:2369] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:06.722575 25087 catalog_manager.cc:5484] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65). New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } }
W20240502 14:15:06.735874 26055 tablet.cc:2369] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:06.742748 21737 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:15:06.744542 21737 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a5cb74a14a8d434e986ac7d988731e8a to finish bootstrapping
I20240502 14:15:06.752276 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c91321c0a57d4bec8eb3bfba60cee06e to finish bootstrapping
I20240502 14:15:06.759842 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1360ad293a3e4ecebf03dbf14cca9cb4 to finish bootstrapping
I20240502 14:15:06.767210 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c69e4233b679437b8eb3b811680fa196 to finish bootstrapping
I20240502 14:15:06.774585 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2bf9b3d134bc4ac79a05d942fc2fb68b to finish bootstrapping
I20240502 14:15:06.782096 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6365836c0f2b428bbeb4d69b9ceda2fe to finish bootstrapping
I20240502 14:15:06.789253 21737 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 103df70e837147d9a992da0e46ece12e to finish bootstrapping
I20240502 14:15:06.795855 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25537
I20240502 14:15:06.796634 25659 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:06.872730 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25537
I20240502 14:15:06.890123 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25667
I20240502 14:15:06.891197 25789 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:06.972746 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25667
I20240502 14:15:06.991669 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25797
I20240502 14:15:06.992697 25919 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:07.065670 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25797
I20240502 14:15:07.081228 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25927
I20240502 14:15:07.082186 26049 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:07.168103 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25927
W20240502 14:15:07.326712 25161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111)
W20240502 14:15:07.328006 25161 consensus_peers.cc:485] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:09.948936 25161 consensus_peers.cc:485] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:15:10.936400 25087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40774:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:10.947501 25208 tablet_service.cc:1467] Processing CreateTablet for tablet 02f1d29d09964aa3bfb918e985c18340 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.948017 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f1d29d09964aa3bfb918e985c18340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.954532 26109 tablet_bootstrap.cc:492] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:10.956269 26109 tablet_bootstrap.cc:654] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.958299 26109 tablet_bootstrap.cc:492] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:10.958487 26109 ts_tablet_manager.cc:1367] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:15:10.958070 25469 tablet_service.cc:1467] Processing CreateTablet for tablet 60e291dc59934bc38239bb84b76975c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.958058 25337 tablet_service.cc:1467] Processing CreateTablet for tablet 02f1d29d09964aa3bfb918e985c18340 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.958889 25338 tablet_service.cc:1467] Processing CreateTablet for tablet 291df3cd98de4152ab6aaf069bc45a36 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.959643 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e291dc59934bc38239bb84b76975c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.959632 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f1d29d09964aa3bfb918e985c18340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.959537 25339 tablet_service.cc:1467] Processing CreateTablet for tablet 60e291dc59934bc38239bb84b76975c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.958214 25468 tablet_service.cc:1467] Processing CreateTablet for tablet 291df3cd98de4152ab6aaf069bc45a36 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.959705 26109 raft_consensus.cc:348] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:10.959884 26109 raft_consensus.cc:374] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.959975 26109 raft_consensus.cc:729] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.960250 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291df3cd98de4152ab6aaf069bc45a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.960280 26109 consensus_queue.cc:260] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:10.960670 25468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291df3cd98de4152ab6aaf069bc45a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.960767 26109 ts_tablet_manager.cc:1398] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:10.962311 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e291dc59934bc38239bb84b76975c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.975813 26111 tablet_bootstrap.cc:492] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:10.977283 26112 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:10.979864 26111 tablet_bootstrap.cc:654] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.980525 26112 tablet_bootstrap.cc:654] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.981112 26111 log.cc:824] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:10.981727 26112 log.cc:824] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:10.983685 26111 tablet_bootstrap.cc:492] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:10.983877 26111 ts_tablet_manager.cc:1367] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240502 14:15:10.984225 26112 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:10.984411 26112 ts_tablet_manager.cc:1367] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:15:10.989190 26111 raft_consensus.cc:348] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:10.989485 26111 raft_consensus.cc:374] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.989585 26111 raft_consensus.cc:729] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.989636 26112 raft_consensus.cc:348] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:10.989935 26112 raft_consensus.cc:374] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.990023 26112 raft_consensus.cc:729] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.990150 26111 consensus_queue.cc:260] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:10.990552 26112 consensus_queue.cc:260] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:10.991494 26111 ts_tablet_manager.cc:1398] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240502 14:15:10.991901 26111 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:10.993011 26112 ts_tablet_manager.cc:1398] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:15:10.993286 26111 tablet_bootstrap.cc:654] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.993296 26112 tablet_bootstrap.cc:492] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:10.994612 26112 tablet_bootstrap.cc:654] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.995581 26111 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:10.995806 26111 ts_tablet_manager.cc:1367] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:15:10.996620 26111 raft_consensus.cc:348] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:10.996789 26111 raft_consensus.cc:374] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.996891 26111 raft_consensus.cc:729] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.997118 26111 consensus_queue.cc:260] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:10.997526 26111 ts_tablet_manager.cc:1398] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:10.997627 26112 tablet_bootstrap.cc:492] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:10.997789 26112 ts_tablet_manager.cc:1367] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:15:10.998600 26112 raft_consensus.cc:348] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:10.998790 26112 raft_consensus.cc:374] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.998885 26112 raft_consensus.cc:729] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.999110 26112 consensus_queue.cc:260] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:10.999498 26112 ts_tablet_manager.cc:1398] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:10.999742 26112 tablet_bootstrap.cc:492] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:11.001237 26112 tablet_bootstrap.cc:654] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.002844 26112 tablet_bootstrap.cc:492] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:11.002956 26112 ts_tablet_manager.cc:1367] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:11.003402 26112 raft_consensus.cc:348] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.003525 26112 raft_consensus.cc:374] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.003600 26112 raft_consensus.cc:729] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.003791 26112 consensus_queue.cc:260] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.004122 26112 ts_tablet_manager.cc:1398] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240502 14:15:11.062341 25535 tablet.cc:2369] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:11.103065 26116 raft_consensus.cc:482] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.103327 26116 raft_consensus.cc:504] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.104672 26116 leader_election.cc:288] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:11.110462 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f1d29d09964aa3bfb918e985c18340" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a" is_pre_election: true
I20240502 14:15:11.110832 25228 raft_consensus.cc:2454] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 0.
I20240502 14:15:11.111342 25293 leader_election.cc:302] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:11.111654 26116 raft_consensus.cc:2781] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.111773 26116 raft_consensus.cc:482] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.111869 26116 raft_consensus.cc:3037] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.114076 26116 raft_consensus.cc:504] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.114475 26116 leader_election.cc:288] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:11.114893 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f1d29d09964aa3bfb918e985c18340" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:11.115044 25228 raft_consensus.cc:3037] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.116892 25228 raft_consensus.cc:2454] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 1.
I20240502 14:15:11.117281 25293 leader_election.cc:302] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:11.117549 26116 raft_consensus.cc:2781] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.118022 26116 raft_consensus.cc:686] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Becoming Leader. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Running, Role: LEADER
I20240502 14:15:11.118510 26116 consensus_queue.cc:237] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.122244 25087 catalog_manager.cc:5484] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: term changed from 0 to 1, leader changed from <none> to c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } }
W20240502 14:15:11.136474 25405 tablet.cc:2369] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:11.155004 26115 raft_consensus.cc:482] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.155222 26115 raft_consensus.cc:504] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:11.156453 26115 leader_election.cc:288] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:11.162050 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e291dc59934bc38239bb84b76975c9" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" is_pre_election: true
I20240502 14:15:11.162345 25359 raft_consensus.cc:2454] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 0.
I20240502 14:15:11.162844 25422 leader_election.cc:302] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:11.163172 26115 raft_consensus.cc:2781] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.163290 26115 raft_consensus.cc:482] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.163379 26115 raft_consensus.cc:3037] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.165200 26115 raft_consensus.cc:504] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:11.165577 26115 leader_election.cc:288] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:11.165990 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e291dc59934bc38239bb84b76975c9" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:11.166136 25359 raft_consensus.cc:3037] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.168036 25359 raft_consensus.cc:2454] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 1.
I20240502 14:15:11.168337 25422 leader_election.cc:302] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:11.168578 26115 raft_consensus.cc:2781] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.168951 26115 raft_consensus.cc:686] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:11.169353 26115 consensus_queue.cc:237] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:11.172685 25087 catalog_manager.cc:5484] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.344676 26116 raft_consensus.cc:482] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.344861 26116 raft_consensus.cc:504] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.345368 26116 leader_election.cc:288] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:11.350433 25489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291df3cd98de4152ab6aaf069bc45a36" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" is_pre_election: true
I20240502 14:15:11.350783 25489 raft_consensus.cc:2454] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 0.
I20240502 14:15:11.351238 25293 leader_election.cc:302] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:11.351501 26116 raft_consensus.cc:2781] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.351583 26116 raft_consensus.cc:482] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.351689 26116 raft_consensus.cc:3037] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.353480 26116 raft_consensus.cc:504] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.353904 26116 leader_election.cc:288] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Requested vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:11.354310 25489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291df3cd98de4152ab6aaf069bc45a36" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
I20240502 14:15:11.354465 25489 raft_consensus.cc:3037] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.356416 25489 raft_consensus.cc:2454] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 1.
I20240502 14:15:11.356762 25293 leader_election.cc:302] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:11.357017 26116 raft_consensus.cc:2781] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.357158 26116 raft_consensus.cc:686] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Becoming Leader. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Running, Role: LEADER
I20240502 14:15:11.357381 26116 consensus_queue.cc:237] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:11.359434 25087 catalog_manager.cc:5484] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: term changed from 0 to 1, leader changed from <none> to c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.601068 26115 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.628295 26116 consensus_queue.cc:1035] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:11.724609 26128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:11.724858 26128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:11.731856 26128 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:15:11.793123 26133 consensus_queue.cc:1035] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 191.787 ms
time per row: 0.0958933 ms
I20240502 14:15:12.051831 25084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40774:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:12.059958 25208 tablet_service.cc:1467] Processing CreateTablet for tablet e30b47ad42124444aadeabc9924e8681 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.060209 25468 tablet_service.cc:1467] Processing CreateTablet for tablet e30b47ad42124444aadeabc9924e8681 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.060428 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30b47ad42124444aadeabc9924e8681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.060622 25468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30b47ad42124444aadeabc9924e8681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.061167 25207 tablet_service.cc:1467] Processing CreateTablet for tablet fcc673d4727a4c049228070d9e2ada05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.061637 25207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc673d4727a4c049228070d9e2ada05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.065196 26182 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:12.065202 25206 tablet_service.cc:1467] Processing CreateTablet for tablet def417ebb8354201bb6c17eed13ebbf7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.065759 25206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def417ebb8354201bb6c17eed13ebbf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.066870 26182 tablet_bootstrap.cc:654] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.067562 25469 tablet_service.cc:1467] Processing CreateTablet for tablet fcc673d4727a4c049228070d9e2ada05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.068202 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc673d4727a4c049228070d9e2ada05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.071943 25339 tablet_service.cc:1467] Processing CreateTablet for tablet def417ebb8354201bb6c17eed13ebbf7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.072324 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def417ebb8354201bb6c17eed13ebbf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.074501 26183 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:12.075155 26182 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:12.075301 26182 ts_tablet_manager.cc:1367] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:15:12.076237 26183 tablet_bootstrap.cc:654] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.076092 26182 raft_consensus.cc:348] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.076378 26182 raft_consensus.cc:374] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.076478 26182 raft_consensus.cc:729] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.076745 26182 consensus_queue.cc:260] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.077896 26182 ts_tablet_manager.cc:1398] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:12.078127 26182 tablet_bootstrap.cc:492] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:12.079672 26182 tablet_bootstrap.cc:654] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.079779 26183 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:12.080734 26186 tablet_bootstrap.cc:492] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:12.079946 26183 ts_tablet_manager.cc:1367] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:15:12.081789 26183 raft_consensus.cc:348] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.081933 26182 tablet_bootstrap.cc:492] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:12.082032 26183 raft_consensus.cc:374] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.082152 26182 ts_tablet_manager.cc:1367] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:15:12.082154 26183 raft_consensus.cc:729] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.082366 26183 consensus_queue.cc:260] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.082911 26183 ts_tablet_manager.cc:1398] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:12.083022 26182 raft_consensus.cc:348] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.083230 26183 tablet_bootstrap.cc:492] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:12.083248 26182 raft_consensus.cc:374] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.083343 26182 raft_consensus.cc:729] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.083555 26182 consensus_queue.cc:260] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.084151 26182 ts_tablet_manager.cc:1398] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:12.084726 26183 tablet_bootstrap.cc:654] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.086424 26186 tablet_bootstrap.cc:654] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.087734 26183 tablet_bootstrap.cc:492] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:12.087893 26183 ts_tablet_manager.cc:1367] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:12.088487 26186 tablet_bootstrap.cc:492] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:12.088625 26186 ts_tablet_manager.cc:1367] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:15:12.088490 26183 raft_consensus.cc:348] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:12.088863 26183 raft_consensus.cc:374] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.089224 26186 raft_consensus.cc:348] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:12.089171 26183 raft_consensus.cc:729] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.089375 26186 raft_consensus.cc:374] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.089463 26186 raft_consensus.cc:729] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.089550 26183 consensus_queue.cc:260] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:12.089650 26186 consensus_queue.cc:260] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:12.090044 26183 ts_tablet_manager.cc:1398] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:12.090209 26186 ts_tablet_manager.cc:1398] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:12.090289 26183 tablet_bootstrap.cc:492] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:12.091780 26183 tablet_bootstrap.cc:654] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.093538 26183 tablet_bootstrap.cc:492] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:12.093698 26183 ts_tablet_manager.cc:1367] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:15:12.094506 26183 raft_consensus.cc:348] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.094692 26183 raft_consensus.cc:374] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.094794 26183 raft_consensus.cc:729] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.095007 26183 consensus_queue.cc:260] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.095481 26183 ts_tablet_manager.cc:1398] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:12.157842 26115 raft_consensus.cc:482] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.157992 26115 raft_consensus.cc:504] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.158524 26115 leader_election.cc:288] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:12.164454 25227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc673d4727a4c049228070d9e2ada05" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a" is_pre_election: true
I20240502 14:15:12.164628 25227 raft_consensus.cc:2454] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 0.
I20240502 14:15:12.164974 25423 leader_election.cc:302] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:12.165211 26115 raft_consensus.cc:2781] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.165313 26115 raft_consensus.cc:482] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.165387 26115 raft_consensus.cc:3037] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.167171 26115 raft_consensus.cc:504] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.167598 26115 leader_election.cc:288] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:12.168007 25227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc673d4727a4c049228070d9e2ada05" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:12.168160 25227 raft_consensus.cc:3037] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.169965 25227 raft_consensus.cc:2454] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 1.
I20240502 14:15:12.170269 25423 leader_election.cc:302] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:12.170497 26115 raft_consensus.cc:2781] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:12.170608 26115 raft_consensus.cc:686] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:12.170821 26115 consensus_queue.cc:237] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.172914 25078 catalog_manager.cc:5484] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } }
W20240502 14:15:12.239560 25161 consensus_peers.cc:485] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:15:12.301280 26098 raft_consensus.cc:482] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.301462 26098 raft_consensus.cc:504] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:12.302117 26098 leader_election.cc:288] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:12.304790 26115 raft_consensus.cc:482] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.304971 26115 raft_consensus.cc:504] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.307144 26115 leader_election.cc:288] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:12.307576 25227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a" is_pre_election: true
I20240502 14:15:12.307811 25227 raft_consensus.cc:2454] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 0.
I20240502 14:15:12.308211 25423 leader_election.cc:302] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:12.308346 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def417ebb8354201bb6c17eed13ebbf7" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" is_pre_election: true
I20240502 14:15:12.308487 26115 raft_consensus.cc:2781] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.308584 25359 raft_consensus.cc:2454] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 0.
I20240502 14:15:12.308614 26115 raft_consensus.cc:482] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.308703 26115 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.308964 25161 leader_election.cc:302] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:12.309208 26098 raft_consensus.cc:2781] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.309300 26098 raft_consensus.cc:482] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.309408 26098 raft_consensus.cc:3037] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.310717 26115 raft_consensus.cc:504] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.311192 26115 leader_election.cc:288] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:12.311367 26098 raft_consensus.cc:504] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:12.311631 25227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:12.311821 25227 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.311826 26098 leader_election.cc:288] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:12.312170 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def417ebb8354201bb6c17eed13ebbf7" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:12.312350 25359 raft_consensus.cc:3037] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.313635 25227 raft_consensus.cc:2454] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 1.
I20240502 14:15:12.313935 25423 leader_election.cc:302] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:12.314133 25359 raft_consensus.cc:2454] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 1.
I20240502 14:15:12.314165 26115 raft_consensus.cc:2781] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:12.314311 26115 raft_consensus.cc:686] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:12.314515 25161 leader_election.cc:302] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:12.314600 26115 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:12.314759 26098 raft_consensus.cc:2781] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:12.315208 26098 raft_consensus.cc:686] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Becoming Leader. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:12.315483 26098 consensus_queue.cc:237] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:12.317585 25078 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:12.317744 25084 catalog_manager.cc:5484] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65). New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:12.638043 26115 consensus_queue.cc:1035] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:12.671005 26199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:12.671334 26199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:12.678680 26199 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:15:12.798560 26125 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.802321 26098 consensus_queue.cc:1035] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:12.852977 25161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111) [suppressed 10 similar messages]
INSERT report
rows total: 2000
time total: 184.733 ms
time per row: 0.0923667 ms
I20240502 14:15:13.000167 25087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40774:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:13.007761 25339 tablet_service.cc:1467] Processing CreateTablet for tablet ead87caf4f9c4695a9c1b8b64852cabf (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.008129 25338 tablet_service.cc:1467] Processing CreateTablet for tablet 57558d7d2f9943dc8fe5fd127f3062e8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.008235 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead87caf4f9c4695a9c1b8b64852cabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.008481 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57558d7d2f9943dc8fe5fd127f3062e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.008817 25468 tablet_service.cc:1467] Processing CreateTablet for tablet 57558d7d2f9943dc8fe5fd127f3062e8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.008893 25469 tablet_service.cc:1467] Processing CreateTablet for tablet ead87caf4f9c4695a9c1b8b64852cabf (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.008888 25208 tablet_service.cc:1467] Processing CreateTablet for tablet a68699dbcae34cfc94f2559e0516e5e8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.009176 25468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57558d7d2f9943dc8fe5fd127f3062e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.009266 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68699dbcae34cfc94f2559e0516e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.009502 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead87caf4f9c4695a9c1b8b64852cabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.011971 25337 tablet_service.cc:1467] Processing CreateTablet for tablet a68699dbcae34cfc94f2559e0516e5e8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.012312 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68699dbcae34cfc94f2559e0516e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.024585 26253 tablet_bootstrap.cc:492] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:13.025373 26254 tablet_bootstrap.cc:492] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:13.025568 26252 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:13.026484 26253 tablet_bootstrap.cc:654] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.027179 26254 tablet_bootstrap.cc:654] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.027269 26252 tablet_bootstrap.cc:654] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.028069 26253 tablet_bootstrap.cc:492] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:13.028203 26253 ts_tablet_manager.cc:1367] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:13.028940 26253 raft_consensus.cc:348] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.029057 26254 tablet_bootstrap.cc:492] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:13.029115 26253 raft_consensus.cc:374] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.029155 26254 ts_tablet_manager.cc:1367] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:13.029251 26253 raft_consensus.cc:729] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.029384 26252 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:13.029512 26252 ts_tablet_manager.cc:1367] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:15:13.029489 26253 consensus_queue.cc:260] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.029794 26254 raft_consensus.cc:348] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.029938 26254 raft_consensus.cc:374] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.030051 26254 raft_consensus.cc:729] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.030232 26254 consensus_queue.cc:260] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.030387 26253 ts_tablet_manager.cc:1398] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:13.030301 26252 raft_consensus.cc:348] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:13.030486 26252 raft_consensus.cc:374] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.030563 26252 raft_consensus.cc:729] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.030653 26254 ts_tablet_manager.cc:1398] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:15:13.030854 26254 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:13.030751 26252 consensus_queue.cc:260] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:13.031180 26253 tablet_bootstrap.cc:492] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:13.031378 26252 ts_tablet_manager.cc:1398] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:13.032167 26254 tablet_bootstrap.cc:654] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.032563 26253 tablet_bootstrap.cc:654] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.035755 26254 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:13.036103 26254 ts_tablet_manager.cc:1367] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:15:13.036893 26254 raft_consensus.cc:348] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:13.037101 26254 raft_consensus.cc:374] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.037184 26254 raft_consensus.cc:729] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.037408 26254 consensus_queue.cc:260] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:13.037842 26254 ts_tablet_manager.cc:1398] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:13.037895 26253 tablet_bootstrap.cc:492] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:13.038079 26253 ts_tablet_manager.cc:1367] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 14:15:13.038116 26254 tablet_bootstrap.cc:492] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:13.038890 26253 raft_consensus.cc:348] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.039065 26253 raft_consensus.cc:374] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.039186 26254 tablet_bootstrap.cc:654] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.039191 26253 raft_consensus.cc:729] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.039434 26253 consensus_queue.cc:260] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.040022 26253 ts_tablet_manager.cc:1398] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:13.044072 26254 tablet_bootstrap.cc:492] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:13.044193 26254 ts_tablet_manager.cc:1367] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 14:15:13.044677 26254 raft_consensus.cc:348] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.044808 26254 raft_consensus.cc:374] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.044867 26254 raft_consensus.cc:729] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.044997 26254 consensus_queue.cc:260] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.045316 26254 ts_tablet_manager.cc:1398] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:13.292459 26115 raft_consensus.cc:482] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.292640 26115 raft_consensus.cc:504] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.293136 26115 leader_election.cc:288] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:13.293738 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead87caf4f9c4695a9c1b8b64852cabf" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" is_pre_election: true
I20240502 14:15:13.293948 25358 raft_consensus.cc:2454] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 0.
I20240502 14:15:13.294312 25422 leader_election.cc:302] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:13.294574 26115 raft_consensus.cc:2781] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.294651 26115 raft_consensus.cc:482] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.294749 26115 raft_consensus.cc:3037] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.296474 26115 raft_consensus.cc:504] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.296861 26115 leader_election.cc:288] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:13.297281 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead87caf4f9c4695a9c1b8b64852cabf" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:13.297453 25358 raft_consensus.cc:3037] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.299401 25358 raft_consensus.cc:2454] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 1.
I20240502 14:15:13.299736 25422 leader_election.cc:302] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:13.299958 26115 raft_consensus.cc:2781] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.300108 26115 raft_consensus.cc:686] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:13.300338 26115 consensus_queue.cc:237] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.302295 25079 catalog_manager.cc:5484] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:13.330138 26098 raft_consensus.cc:482] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.330260 26098 raft_consensus.cc:504] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:13.330600 26098 leader_election.cc:288] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:13.330974 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" is_pre_election: true
I20240502 14:15:13.331177 25358 raft_consensus.cc:2454] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 0.
I20240502 14:15:13.331502 25161 leader_election.cc:302] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:13.331724 26098 raft_consensus.cc:2781] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.331822 26098 raft_consensus.cc:482] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.331912 26098 raft_consensus.cc:3037] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.333550 26098 raft_consensus.cc:504] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:13.333906 26098 leader_election.cc:288] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:13.334249 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:13.334389 25358 raft_consensus.cc:3037] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.336050 25358 raft_consensus.cc:2454] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 1.
I20240502 14:15:13.336386 25161 leader_election.cc:302] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:13.336603 26098 raft_consensus.cc:2781] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.336745 26098 raft_consensus.cc:686] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Becoming Leader. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:13.336992 26098 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:13.339068 25079 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65). New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:13.375315 26115 raft_consensus.cc:482] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.375435 26115 raft_consensus.cc:504] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.375838 26115 leader_election.cc:288] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:13.376250 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57558d7d2f9943dc8fe5fd127f3062e8" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" is_pre_election: true
I20240502 14:15:13.376397 25358 raft_consensus.cc:2454] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 0.
I20240502 14:15:13.376711 25422 leader_election.cc:302] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:13.376960 26115 raft_consensus.cc:2781] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.377067 26115 raft_consensus.cc:482] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.377167 26115 raft_consensus.cc:3037] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.379251 26115 raft_consensus.cc:504] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.379617 26115 leader_election.cc:288] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:13.380095 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57558d7d2f9943dc8fe5fd127f3062e8" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:13.380273 25358 raft_consensus.cc:3037] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.382107 25358 raft_consensus.cc:2454] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 1.
I20240502 14:15:13.382437 25422 leader_election.cc:302] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:13.382711 26115 raft_consensus.cc:2781] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.382826 26115 raft_consensus.cc:686] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:13.383064 26115 consensus_queue.cc:237] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:13.385121 25079 catalog_manager.cc:5484] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:13.693156 26115 consensus_queue.cc:1035] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:13.780328 26264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:13.780584 26264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:13.787657 26264 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:15:13.889051 26098 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.926715 25358 raft_consensus.cc:1261] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.927470 26267 consensus_queue.cc:1035] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.935140 26302 mvcc.cc:205] Tried to move back new op lower bound from 7023244549838315520 to 7023244547618361344. Current Snapshot: MvccSnapshot[applied={T|T < 7023244549838315520}]
I20240502 14:15:13.946000 26304 mvcc.cc:205] Tried to move back new op lower bound from 7023244549838315520 to 7023244547618361344. Current Snapshot: MvccSnapshot[applied={T|T < 7023244549838315520}]
INSERT report
rows total: 2000
time total: 181.236 ms
time per row: 0.0906178 ms
I20240502 14:15:14.101135 25079 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40774:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:14.109005 25468 tablet_service.cc:1467] Processing CreateTablet for tablet 503e0727a7a14863847f375032cb8144 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.109210 25208 tablet_service.cc:1467] Processing CreateTablet for tablet 503e0727a7a14863847f375032cb8144 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.109463 25468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503e0727a7a14863847f375032cb8144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.109589 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503e0727a7a14863847f375032cb8144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.110116 25338 tablet_service.cc:1467] Processing CreateTablet for tablet 37ae6a3602fb4437a0e7c337494ea8e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.110530 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ae6a3602fb4437a0e7c337494ea8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.110687 25469 tablet_service.cc:1467] Processing CreateTablet for tablet 0723f00c11d744f8b244a3d003ce7b7a (DEFAULT_TABLE table=rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.111099 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723f00c11d744f8b244a3d003ce7b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.113621 25206 tablet_service.cc:1467] Processing CreateTablet for tablet 37ae6a3602fb4437a0e7c337494ea8e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.114007 25206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ae6a3602fb4437a0e7c337494ea8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.115438 25207 tablet_service.cc:1467] Processing CreateTablet for tablet 0723f00c11d744f8b244a3d003ce7b7a (DEFAULT_TABLE table=rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.115779 25207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723f00c11d744f8b244a3d003ce7b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.122298 26320 tablet_bootstrap.cc:492] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:14.123761 26321 tablet_bootstrap.cc:492] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:14.124468 26320 tablet_bootstrap.cc:654] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.126487 26321 tablet_bootstrap.cc:654] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.130733 26322 tablet_bootstrap.cc:492] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:14.130901 26320 tablet_bootstrap.cc:492] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:14.131045 26321 tablet_bootstrap.cc:492] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:14.131057 26320 ts_tablet_manager.cc:1367] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240502 14:15:14.131182 26321 ts_tablet_manager.cc:1367] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240502 14:15:14.131664 26320 raft_consensus.cc:348] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.131915 26320 raft_consensus.cc:374] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.132009 26320 raft_consensus.cc:729] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.132000 26321 raft_consensus.cc:348] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.132197 26321 raft_consensus.cc:374] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.132297 26321 raft_consensus.cc:729] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.132254 26320 consensus_queue.cc:260] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.132490 26322 tablet_bootstrap.cc:654] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.132534 26321 consensus_queue.cc:260] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.132833 26320 ts_tablet_manager.cc:1398] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:14.133129 26321 ts_tablet_manager.cc:1398] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:14.133425 26321 tablet_bootstrap.cc:492] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:14.133131 26320 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:14.135251 26321 tablet_bootstrap.cc:654] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.135855 26320 tablet_bootstrap.cc:654] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.137722 26320 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:14.137876 26320 ts_tablet_manager.cc:1367] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:14.138291 26321 tablet_bootstrap.cc:492] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:14.138450 26322 tablet_bootstrap.cc:492] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:14.138454 26321 ts_tablet_manager.cc:1367] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 14:15:14.138571 26322 ts_tablet_manager.cc:1367] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:15:14.138605 26320 raft_consensus.cc:348] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.138787 26320 raft_consensus.cc:374] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.138864 26320 raft_consensus.cc:729] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.139112 26320 consensus_queue.cc:260] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.139155 26321 raft_consensus.cc:348] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.139340 26321 raft_consensus.cc:374] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.139410 26321 raft_consensus.cc:729] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.139263 26322 raft_consensus.cc:348] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.139628 26322 raft_consensus.cc:374] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.139637 26320 ts_tablet_manager.cc:1398] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:14.139714 26322 raft_consensus.cc:729] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.139657 26321 consensus_queue.cc:260] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.140095 26321 ts_tablet_manager.cc:1398] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:14.140061 26322 consensus_queue.cc:260] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.140383 26321 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:14.140767 26322 ts_tablet_manager.cc:1398] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:14.141783 26321 tablet_bootstrap.cc:654] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.144026 26321 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:14.144260 26321 ts_tablet_manager.cc:1367] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:15:14.144852 26321 raft_consensus.cc:348] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.145021 26321 raft_consensus.cc:374] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.145129 26321 raft_consensus.cc:729] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.145345 26321 consensus_queue.cc:260] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.145783 26321 ts_tablet_manager.cc:1398] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:14.270253 26295 raft_consensus.cc:482] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.270406 26295 raft_consensus.cc:504] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.270999 26295 leader_election.cc:288] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:14.275739 25489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503e0727a7a14863847f375032cb8144" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" is_pre_election: true
I20240502 14:15:14.275962 25489 raft_consensus.cc:2454] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 0.
I20240502 14:15:14.276301 25162 leader_election.cc:302] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:14.276548 26295 raft_consensus.cc:2781] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.276640 26295 raft_consensus.cc:482] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.276717 26295 raft_consensus.cc:3037] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.278616 26295 raft_consensus.cc:504] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.279011 26295 leader_election.cc:288] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Requested vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:14.279417 25489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503e0727a7a14863847f375032cb8144" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
I20240502 14:15:14.279583 25489 raft_consensus.cc:3037] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.281296 25489 raft_consensus.cc:2454] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 1.
I20240502 14:15:14.281600 25162 leader_election.cc:302] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:14.281857 26295 raft_consensus.cc:2781] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.281986 26295 raft_consensus.cc:686] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Becoming Leader. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:14.282220 26295 consensus_queue.cc:237] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.284243 25082 catalog_manager.cc:5484] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65). New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:14.300113 26269 raft_consensus.cc:482] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.300221 26269 raft_consensus.cc:504] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.300604 26269 leader_election.cc:288] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:14.300980 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723f00c11d744f8b244a3d003ce7b7a" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a" is_pre_election: true
I20240502 14:15:14.301136 25228 raft_consensus.cc:2454] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 0.
I20240502 14:15:14.301445 25423 leader_election.cc:302] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:14.301671 26269 raft_consensus.cc:2781] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.301769 26269 raft_consensus.cc:482] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.301831 26269 raft_consensus.cc:3037] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.303565 26269 raft_consensus.cc:504] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.303882 26269 leader_election.cc:288] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:14.304270 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723f00c11d744f8b244a3d003ce7b7a" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:14.304430 25228 raft_consensus.cc:3037] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.306378 25228 raft_consensus.cc:2454] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 1.
I20240502 14:15:14.306675 25423 leader_election.cc:302] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:14.306918 26269 raft_consensus.cc:2781] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.307037 26269 raft_consensus.cc:686] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:14.307286 26269 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.309320 25082 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:14.315994 26172 raft_consensus.cc:482] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.316124 26172 raft_consensus.cc:504] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.316591 26172 leader_election.cc:288] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:14.316962 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ae6a3602fb4437a0e7c337494ea8e8" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a" is_pre_election: true
I20240502 14:15:14.317097 25228 raft_consensus.cc:2454] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 0.
I20240502 14:15:14.317433 25293 leader_election.cc:302] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:14.317673 26172 raft_consensus.cc:2781] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.317775 26172 raft_consensus.cc:482] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.317855 26172 raft_consensus.cc:3037] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.319597 26172 raft_consensus.cc:504] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.319938 26172 leader_election.cc:288] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:14.320317 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ae6a3602fb4437a0e7c337494ea8e8" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:14.320434 25228 raft_consensus.cc:3037] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.322021 25228 raft_consensus.cc:2454] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 1.
I20240502 14:15:14.322357 25293 leader_election.cc:302] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:14.322585 26172 raft_consensus.cc:2781] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.322958 26172 raft_consensus.cc:686] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Becoming Leader. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Running, Role: LEADER
I20240502 14:15:14.323259 26172 consensus_queue.cc:237] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:14.325310 25082 catalog_manager.cc:5484] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: term changed from 0 to 1, leader changed from <none> to c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:14.639608 25161 consensus_peers.cc:485] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:15:14.660313 26335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:14.660552 26335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:14.666918 26335 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:15:14.695914 26269 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 14:15:14.785805 25228 raft_consensus.cc:1261] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.786676 26172 consensus_queue.cc:1035] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.802359 25489 raft_consensus.cc:1261] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.802961 26295 consensus_queue.cc:1035] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.819752 26348 mvcc.cc:205] Tried to move back new op lower bound from 7023244553355124736 to 7023244551469395968. Current Snapshot: MvccSnapshot[applied={T|T < 7023244553355124736}]
INSERT report
rows total: 2000
time total: 207.542 ms
time per row: 0.103771 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:15:15.901324 25079 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40774:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:15.909143 25338 tablet_service.cc:1467] Processing CreateTablet for tablet 537b91a6ab124022a1cdfb077055d06f (DEFAULT_TABLE table=rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.909457 25469 tablet_service.cc:1467] Processing CreateTablet for tablet 537b91a6ab124022a1cdfb077055d06f (DEFAULT_TABLE table=rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.909601 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b91a6ab124022a1cdfb077055d06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.909703 25337 tablet_service.cc:1467] Processing CreateTablet for tablet 232f776da8074d5bb2643f4add1cb06a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.909827 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b91a6ab124022a1cdfb077055d06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.910022 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232f776da8074d5bb2643f4add1cb06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.913283 25339 tablet_service.cc:1467] Processing CreateTablet for tablet 66c85e449bea4f829fe6d1c79206591d (DEFAULT_TABLE table=rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.913506 25207 tablet_service.cc:1467] Processing CreateTablet for tablet 66c85e449bea4f829fe6d1c79206591d (DEFAULT_TABLE table=rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.913637 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c85e449bea4f829fe6d1c79206591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.913897 25207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c85e449bea4f829fe6d1c79206591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.915453 25468 tablet_service.cc:1467] Processing CreateTablet for tablet 232f776da8074d5bb2643f4add1cb06a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.915808 25468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232f776da8074d5bb2643f4add1cb06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.924836 26394 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:15.926617 26394 tablet_bootstrap.cc:654] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.928869 26396 tablet_bootstrap.cc:492] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a: Bootstrap starting.
I20240502 14:15:15.929438 26394 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:15.929563 26394 ts_tablet_manager.cc:1367] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:15:15.930586 26396 tablet_bootstrap.cc:654] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.930439 26394 raft_consensus.cc:348] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.930778 26395 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:15.930802 26394 raft_consensus.cc:374] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.930935 26394 raft_consensus.cc:729] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.931216 26394 consensus_queue.cc:260] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.931739 26394 ts_tablet_manager.cc:1398] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:15.932034 26394 tablet_bootstrap.cc:492] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:15.932662 26395 tablet_bootstrap.cc:654] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.933492 26394 tablet_bootstrap.cc:654] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.938642 26395 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:15.938645 26396 tablet_bootstrap.cc:492] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a: No bootstrap required, opened a new log
I20240502 14:15:15.938773 26396 ts_tablet_manager.cc:1367] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240502 14:15:15.938783 26395 ts_tablet_manager.cc:1367] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:15:15.939345 26395 raft_consensus.cc:348] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.939364 26396 raft_consensus.cc:348] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:15.939488 26396 raft_consensus.cc:374] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.939483 26395 raft_consensus.cc:374] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.939581 26395 raft_consensus.cc:729] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.939581 26396 raft_consensus.cc:729] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.939769 26396 consensus_queue.cc:260] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:15.939788 26395 consensus_queue.cc:260] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.940259 26395 ts_tablet_manager.cc:1398] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:15:15.940269 26396 ts_tablet_manager.cc:1398] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:15.940569 26395 tablet_bootstrap.cc:492] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4: Bootstrap starting.
I20240502 14:15:15.941136 26394 tablet_bootstrap.cc:492] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:15.941303 26394 ts_tablet_manager.cc:1367] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240502 14:15:15.942044 26395 tablet_bootstrap.cc:654] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.942025 26394 raft_consensus.cc:348] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.942242 26394 raft_consensus.cc:374] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.942342 26394 raft_consensus.cc:729] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.942544 26394 consensus_queue.cc:260] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.943039 26394 ts_tablet_manager.cc:1398] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:15.943354 26394 tablet_bootstrap.cc:492] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e: Bootstrap starting.
I20240502 14:15:15.944653 26394 tablet_bootstrap.cc:654] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.946274 26394 tablet_bootstrap.cc:492] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e: No bootstrap required, opened a new log
I20240502 14:15:15.946456 26394 ts_tablet_manager.cc:1367] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:15.946337 26395 tablet_bootstrap.cc:492] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4: No bootstrap required, opened a new log
I20240502 14:15:15.947141 26395 ts_tablet_manager.cc:1367] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:15:15.947292 26394 raft_consensus.cc:348] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:15.947557 26394 raft_consensus.cc:374] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.947717 26394 raft_consensus.cc:729] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.947925 26395 raft_consensus.cc:348] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.948055 26394 consensus_queue.cc:260] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:15.948254 26395 raft_consensus.cc:374] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.948367 26395 raft_consensus.cc:729] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.948551 26394 ts_tablet_manager.cc:1398] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:15.948611 26395 consensus_queue.cc:260] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:15.949067 26395 ts_tablet_manager.cc:1398] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:16.035693 26172 raft_consensus.cc:482] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.035845 26172 raft_consensus.cc:504] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:16.036334 26172 leader_election.cc:288] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:16.036823 25226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c85e449bea4f829fe6d1c79206591d" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a" is_pre_election: true
I20240502 14:15:16.037019 25226 raft_consensus.cc:2454] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 0.
I20240502 14:15:16.037442 25293 leader_election.cc:302] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:16.037714 26172 raft_consensus.cc:2781] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.037819 26172 raft_consensus.cc:482] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.037904 26172 raft_consensus.cc:3037] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.039748 26172 raft_consensus.cc:504] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:16.040105 26172 leader_election.cc:288] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:16.040566 25226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c85e449bea4f829fe6d1c79206591d" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:16.040736 25226 raft_consensus.cc:3037] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.042470 25226 raft_consensus.cc:2454] T 66c85e449bea4f829fe6d1c79206591d P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 1.
I20240502 14:15:16.042812 25293 leader_election.cc:302] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:16.043088 26172 raft_consensus.cc:2781] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.043430 26172 raft_consensus.cc:686] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Becoming Leader. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Running, Role: LEADER
I20240502 14:15:16.043727 26172 consensus_queue.cc:237] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } }
I20240502 14:15:16.045945 25079 catalog_manager.cc:5484] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: term changed from 0 to 1, leader changed from <none> to c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:16.046440 26172 raft_consensus.cc:482] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.046573 26172 raft_consensus.cc:504] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:16.046940 26172 leader_election.cc:288] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:16.047459 25489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b91a6ab124022a1cdfb077055d06f" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" is_pre_election: true
I20240502 14:15:16.047737 25489 raft_consensus.cc:2454] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 0.
I20240502 14:15:16.048172 25293 leader_election.cc:302] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:16.048506 26172 raft_consensus.cc:2781] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.048600 26172 raft_consensus.cc:482] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.048709 26172 raft_consensus.cc:3037] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.050385 26172 raft_consensus.cc:504] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:16.050731 26172 leader_election.cc:288] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Requested vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:16.051168 25489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b91a6ab124022a1cdfb077055d06f" candidate_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
I20240502 14:15:16.051345 25489 raft_consensus.cc:3037] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.053102 25489 raft_consensus.cc:2454] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91321c0a57d4bec8eb3bfba60cee06e in term 1.
I20240502 14:15:16.053447 25293 leader_election.cc:302] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:16.053678 26172 raft_consensus.cc:2781] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.053846 26172 raft_consensus.cc:686] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Becoming Leader. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Running, Role: LEADER
I20240502 14:15:16.054051 26172 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:16.056227 25079 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: term changed from 0 to 1, leader changed from <none> to c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.099769 26267 raft_consensus.cc:482] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.099958 26267 raft_consensus.cc:504] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:16.100399 26267 leader_election.cc:288] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:16.100899 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232f776da8074d5bb2643f4add1cb06a" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" is_pre_election: true
I20240502 14:15:16.101081 25359 raft_consensus.cc:2454] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 0.
I20240502 14:15:16.101477 25422 leader_election.cc:302] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:16.101727 26267 raft_consensus.cc:2781] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.101835 26267 raft_consensus.cc:482] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.101954 26267 raft_consensus.cc:3037] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.103857 26267 raft_consensus.cc:504] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:16.104249 26267 leader_election.cc:288] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:16.104681 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232f776da8074d5bb2643f4add1cb06a" candidate_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:16.104842 25359 raft_consensus.cc:3037] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.106523 25359 raft_consensus.cc:2454] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1360ad293a3e4ecebf03dbf14cca9cb4 in term 1.
I20240502 14:15:16.106837 25422 leader_election.cc:302] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1360ad293a3e4ecebf03dbf14cca9cb4, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:16.107107 26267 raft_consensus.cc:2781] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.107450 26267 raft_consensus.cc:686] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:16.107694 26267 consensus_queue.cc:237] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } }
I20240502 14:15:16.109756 25079 catalog_manager.cc:5484] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } }
W20240502 14:15:16.478742 26408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:16.479071 26408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:16.485541 26408 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:15:16.597604 26172 consensus_queue.cc:1035] T 66c85e449bea4f829fe6d1c79206591d P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.608386 26267 consensus_queue.cc:1035] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.618788 25489 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:16.619992 26441 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 203.492 ms
time per row: 0.101746 ms
I20240502 14:15:16.818775 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:43117
--local_ip_for_outbound_sockets=127.21.58.68
--tserver_master_addrs=127.21.58.126:40155
--webserver_port=35987
--webserver_interface=127.21.58.68
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:15:17.133316 26462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:17.133622 26462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:17.133769 26462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:17.141060 26462 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:17.141160 26462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:17.141269 26462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:15:17.153410 26462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:43117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=35987
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:17.154500 26462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:17.156080 26462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:17.167346 26470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.167999 26468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.168565 26467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:17.169222 26462 server_base.cc:1034] running on GCE node
I20240502 14:15:17.169679 26462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:17.170413 26462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:17.171594 26462 hybrid_clock.cc:648] HybridClock initialized: now 1714659317171553 us; error 34 us; skew 500 ppm
I20240502 14:15:17.171931 26462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:17.173775 26462 webserver.cc:460] Webserver started at http://127.21.58.68:35987/ using document root <none> and password file <none>
I20240502 14:15:17.174355 26462 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:17.174463 26462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:17.178846 26462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:17.181254 26476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.182366 26462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:17.182497 26462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c69e4233b679437b8eb3b811680fa196"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:17.183318 26462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 14:15:17.222297 25161 consensus_peers.cc:485] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:15:17.242657 26462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:17.243408 26462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:17.243661 26462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:17.244508 26462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:17.245893 26462 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:17.245960 26462 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.246047 26462 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:17.246088 26462 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.275663 26462 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:43117
I20240502 14:15:17.275693 26590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:43117 every 8 connection(s)
I20240502 14:15:17.276930 26462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:15:17.282827 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 26462
I20240502 14:15:17.283442 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:44755
--local_ip_for_outbound_sockets=127.21.58.69
--tserver_master_addrs=127.21.58.126:40155
--webserver_port=43017
--webserver_interface=127.21.58.69
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:17.288512 26591 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:17.288815 26591 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:17.289777 26591 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:17.291316 25080 ts_manager.cc:194] Re-registered known tserver with Master: c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117)
I20240502 14:15:17.292431 25080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:38631
W20240502 14:15:17.592460 26595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:17.592705 26595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:17.592818 26595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:17.600327 26595 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:17.600423 26595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:17.600574 26595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:15:17.608251 26595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:44755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=43017
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:17.609323 26595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:17.610810 26595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:17.621775 26601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.622347 26603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.622817 26600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:17.623478 26595 server_base.cc:1034] running on GCE node
I20240502 14:15:17.623939 26595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:17.624876 26595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:17.626080 26595 hybrid_clock.cc:648] HybridClock initialized: now 1714659317626040 us; error 32 us; skew 500 ppm
I20240502 14:15:17.626331 26595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:17.628329 26595 webserver.cc:460] Webserver started at http://127.21.58.69:43017/ using document root <none> and password file <none>
I20240502 14:15:17.628942 26595 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:17.629050 26595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:17.633689 26595 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:15:17.636198 26609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.637337 26595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:17.637465 26595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:17.638171 26595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:17.663111 26595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:17.663688 26595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:17.663900 26595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:17.664598 26595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:17.665899 26595 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:17.665997 26595 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.666082 26595 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:17.666146 26595 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.696190 26595 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:44755
I20240502 14:15:17.696233 26721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:44755 every 8 connection(s)
I20240502 14:15:17.697393 26595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:15:17.707427 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 26595
I20240502 14:15:17.708065 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.70:35761
--local_ip_for_outbound_sockets=127.21.58.70
--tserver_master_addrs=127.21.58.126:40155
--webserver_port=40483
--webserver_interface=127.21.58.70
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:17.708135 26722 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:17.708389 26722 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:17.709070 26722 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:17.711390 25080 ts_manager.cc:194] Re-registered known tserver with Master: 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755)
I20240502 14:15:17.712296 25080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:46533
W20240502 14:15:18.020211 26725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:18.020457 26725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:18.020601 26725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:18.027441 26725 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:18.027591 26725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:18.027741 26725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.70
I20240502 14:15:18.036427 26725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.70:35761
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.58.70
--webserver_port=40483
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.70
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:18.037653 26725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:18.039343 26725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:18.051013 26730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:18.051563 26731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:18.052122 26733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:18.052738 26725 server_base.cc:1034] running on GCE node
I20240502 14:15:18.053251 26725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:18.054065 26725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:18.055245 26725 hybrid_clock.cc:648] HybridClock initialized: now 1714659318055195 us; error 40 us; skew 500 ppm
I20240502 14:15:18.055517 26725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:18.057476 26725 webserver.cc:460] Webserver started at http://127.21.58.70:40483/ using document root <none> and password file <none>
I20240502 14:15:18.058161 26725 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:18.058281 26725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:18.062924 26725 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:18.065358 26739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:18.066452 26725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.066592 26725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6365836c0f2b428bbeb4d69b9ceda2fe"
format_stamp: "Formatted at 2024-05-02 14:15:05 on dist-test-slave-wtgm"
I20240502 14:15:18.067339 26725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:18.079804 26725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:18.080420 26725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:18.080660 26725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:18.081418 26725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:18.082690 26725 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:18.082762 26725 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:18.082876 26725 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:18.082928 26725 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:18.112357 26725 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.70:35761
I20240502 14:15:18.112437 26851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.70:35761 every 8 connection(s)
I20240502 14:15:18.113749 26725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:15:18.120529 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 26725
I20240502 14:15:18.121212 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.71:37231
--local_ip_for_outbound_sockets=127.21.58.71
--tserver_master_addrs=127.21.58.126:40155
--webserver_port=39153
--webserver_interface=127.21.58.71
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:18.125101 26852 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:18.125402 26852 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:18.126291 26852 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:18.127858 25080 ts_manager.cc:194] Re-registered known tserver with Master: 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70:35761)
I20240502 14:15:18.128649 25080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.70:36603
W20240502 14:15:18.283509 25161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.71:37231: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:15:18.294514 26591 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
W20240502 14:15:18.432739 26855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:18.432971 26855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:18.433090 26855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:18.440062 26855 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:18.440160 26855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:18.440305 26855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.71
I20240502 14:15:18.448884 26855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.71:37231
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.58.71
--webserver_port=39153
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.71
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:18.450127 26855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:18.451746 26855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:18.463433 26863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:18.463868 26865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:18.464434 26862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:18.465081 26855 server_base.cc:1034] running on GCE node
I20240502 14:15:18.465528 26855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:18.466265 26855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:18.467446 26855 hybrid_clock.cc:648] HybridClock initialized: now 1714659318467405 us; error 31 us; skew 500 ppm
I20240502 14:15:18.467728 26855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:18.469651 26855 webserver.cc:460] Webserver started at http://127.21.58.71:39153/ using document root <none> and password file <none>
I20240502 14:15:18.470335 26855 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:18.470453 26855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:18.474987 26855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:18.477516 26871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:18.478742 26855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.478888 26855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "103df70e837147d9a992da0e46ece12e"
format_stamp: "Formatted at 2024-05-02 14:15:06 on dist-test-slave-wtgm"
I20240502 14:15:18.479667 26855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:18.502918 26855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:18.503576 26855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:18.503858 26855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:18.504637 26855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:18.506242 26878 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 14:15:18.510855 26855 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 14:15:18.510941 26855 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20240502 14:15:18.511096 26855 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 14:15:18.513725 26855 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 14:15:18.513832 26855 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240502 14:15:18.514101 26878 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Bootstrap starting.
I20240502 14:15:18.530843 26878 log.cc:824] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:18.533329 26878 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:18.533752 26878 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Bootstrap complete.
I20240502 14:15:18.534821 26878 ts_tablet_manager.cc:1367] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20240502 14:15:18.545655 26878 raft_consensus.cc:348] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:18.546659 26878 raft_consensus.cc:729] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103df70e837147d9a992da0e46ece12e, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.547602 26878 consensus_queue.cc:260] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } }
I20240502 14:15:18.549047 26878 ts_tablet_manager.cc:1398] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20240502 14:15:18.557993 26855 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.71:37231
I20240502 14:15:18.558053 26986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.71:37231 every 8 connection(s)
I20240502 14:15:18.559178 26855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240502 14:15:18.560276 26988 tablet.cc:2369] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:18.564961 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 26855
I20240502 14:15:18.565251 21737 test_util.cc:268] Using random seed: 999819615
I20240502 14:15:18.569703 26987 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:18.570000 26987 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:18.570797 26987 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:18.573133 25080 ts_manager.cc:194] Re-registered known tserver with Master: 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231)
I20240502 14:15:18.574555 25080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.71:35619
I20240502 14:15:18.580245 21737 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 14:15:18.580648 21737 test_util.cc:268] Using random seed: 999835014
I20240502 14:15:18.596190 21737 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 14:15:18.599331 21737 test_util.cc:268] Using random seed: 999853686
I20240502 14:15:18.632236 21737 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 14:15:18.646773 21737 test_util.cc:268] Using random seed: 999901131
I20240502 14:15:18.695497 21737 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 14:15:18.709105 21737 test_util.cc:268] Using random seed: 999963446
I20240502 14:15:18.714819 26722 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
I20240502 14:15:18.793846 26987 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
I20240502 14:15:18.812775 21737 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 14:15:19.130875 26852 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
I20240502 14:15:20.035830 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:20.038686 26657 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:20.107784 25207 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:20.108493 26787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:20.135062 25469 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:20.206094 26526 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:20.232473 25339 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:21.044410 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:21.065627 26787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:21.093252 25207 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:21.144299 26657 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:21.146997 25469 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:21.196339 26526 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:21.201308 25339 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:21.377410 25357 consensus_queue.cc:237] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:21.380210 25226 raft_consensus.cc:1261] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240502 14:15:21.383157 27103 consensus_queue.cc:1035] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20240502 14:15:21.388965 27169 raft_consensus.cc:2932] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:21.393922 25227 raft_consensus.cc:2932] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:21.395777 25082 catalog_manager.cc:5484] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index -1 to 62, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:21.424978 25295 consensus_peers.cc:485] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e -> Peer 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755): Couldn't send request to peer 2bf9b3d134bc4ac79a05d942fc2fb68b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37ae6a3602fb4437a0e7c337494ea8e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.463088 25488 consensus_queue.cc:237] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:21.478649 25359 raft_consensus.cc:1261] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 14:15:21.479962 27033 consensus_queue.cc:1035] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 14:15:21.484786 27090 raft_consensus.cc:2932] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
I20240502 14:15:21.500133 25080 catalog_manager.cc:5484] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index -1 to 74, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:21.502558 25359 raft_consensus.cc:2932] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
W20240502 14:15:21.509320 25422 consensus_peers.cc:485] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70:35761): Couldn't send request to peer 6365836c0f2b428bbeb4d69b9ceda2fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57558d7d2f9943dc8fe5fd127f3062e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.583842 25488 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:21.586092 25226 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240502 14:15:21.586768 27045 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240502 14:15:21.589599 27045 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:21.591689 25227 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
W20240502 14:15:21.597162 25425 consensus_peers.cc:485] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117): Couldn't send request to peer c69e4233b679437b8eb3b811680fa196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e30b47ad42124444aadeabc9924e8681. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.598554 25080 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index -1 to 57, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:21.681185 25487 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:21.683917 25226 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240502 14:15:21.684720 27090 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 14:15:21.689550 27090 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } } }
I20240502 14:15:21.693672 25227 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } } }
I20240502 14:15:21.703022 25082 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index -1 to 71, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } } }
W20240502 14:15:21.721992 25422 consensus_peers.cc:485] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0723f00c11d744f8b244a3d003ce7b7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.828958 25489 consensus_queue.cc:237] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
W20240502 14:15:21.838925 25425 consensus_peers.cc:485] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755): Couldn't send request to peer 2bf9b3d134bc4ac79a05d942fc2fb68b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60e291dc59934bc38239bb84b76975c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.847808 25357 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 14:15:21.849391 27040 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240502 14:15:21.862325 27090 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:21.864315 25356 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:21.871145 25082 catalog_manager.cc:5484] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index -1 to 88, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:21.968432 27232 ts_tablet_manager.cc:902] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Initiating tablet copy from peer c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:21.973217 27232 tablet_copy_client.cc:308] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44323
I20240502 14:15:21.980732 25226 consensus_queue.cc:237] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:21.983253 25488 raft_consensus.cc:1261] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240502 14:15:21.984112 27057 consensus_queue.cc:1035] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 14:15:21.989951 27222 raft_consensus.cc:2932] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
I20240502 14:15:21.990785 25489 raft_consensus.cc:2932] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
I20240502 14:15:22.007115 25080 catalog_manager.cc:5484] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index -1 to 76, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:22.023212 25293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.65:46045, user_credentials={real_user=slave}} blocked reactor thread for 46577.9us
W20240502 14:15:22.025974 25161 consensus_peers.cc:485] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70:35761): Couldn't send request to peer 6365836c0f2b428bbeb4d69b9ceda2fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 503e0727a7a14863847f375032cb8144. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.039899 25379 tablet_copy_service.cc:140] P c91321c0a57d4bec8eb3bfba60cee06e: Received BeginTabletCopySession request for tablet 37ae6a3602fb4437a0e7c337494ea8e8 from peer 2bf9b3d134bc4ac79a05d942fc2fb68b ({username='slave'} at 127.21.58.69:35541)
I20240502 14:15:22.040221 25379 tablet_copy_service.cc:161] P c91321c0a57d4bec8eb3bfba60cee06e: Beginning new tablet copy session on tablet 37ae6a3602fb4437a0e7c337494ea8e8 from peer 2bf9b3d134bc4ac79a05d942fc2fb68b at {username='slave'} at 127.21.58.69:35541: session id = 2bf9b3d134bc4ac79a05d942fc2fb68b-37ae6a3602fb4437a0e7c337494ea8e8
I20240502 14:15:22.042958 25379 tablet_copy_source_session.cc:205] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.049054 27232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ae6a3602fb4437a0e7c337494ea8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.057720 27243 ts_tablet_manager.cc:902] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:22.078544 25224 consensus_queue.cc:237] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:22.081791 27248 ts_tablet_manager.cc:902] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:22.081841 26941 raft_consensus.cc:1261] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:22.083140 27151 consensus_queue.cc:1035] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:22.095109 25164 consensus_peers.cc:485] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a -> Peer c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117): Couldn't send request to peer c69e4233b679437b8eb3b811680fa196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de14694a11e14f6082bba7fa70f51553. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.105022 27243 tablet_copy_client.cc:308] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:22.152173 27057 raft_consensus.cc:2932] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:22.156628 26940 raft_consensus.cc:2932] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:22.161538 27259 ts_tablet_manager.cc:902] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:22.177567 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet 57558d7d2f9943dc8fe5fd127f3062e8 from peer 6365836c0f2b428bbeb4d69b9ceda2fe ({username='slave'} at 127.21.58.70:60527)
I20240502 14:15:22.177914 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet 57558d7d2f9943dc8fe5fd127f3062e8 from peer 6365836c0f2b428bbeb4d69b9ceda2fe at {username='slave'} at 127.21.58.70:60527: session id = 6365836c0f2b428bbeb4d69b9ceda2fe-57558d7d2f9943dc8fe5fd127f3062e8
I20240502 14:15:22.184895 25509 tablet_copy_source_session.cc:205] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.190891 27243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57558d7d2f9943dc8fe5fd127f3062e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.204574 27243 tablet_copy_client.cc:783] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.208649 27243 tablet_copy_client.cc:647] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.210814 25080 catalog_manager.cc:5484] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index -1 to 2, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.240780 27243 tablet_copy_client.cc:523] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:22.248600 27243 tablet_bootstrap.cc:492] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap starting.
I20240502 14:15:22.265627 27248 tablet_copy_client.cc:308] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:22.291872 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet e30b47ad42124444aadeabc9924e8681 from peer c69e4233b679437b8eb3b811680fa196 ({username='slave'} at 127.21.58.68:40165)
I20240502 14:15:22.292100 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet e30b47ad42124444aadeabc9924e8681 from peer c69e4233b679437b8eb3b811680fa196 at {username='slave'} at 127.21.58.68:40165: session id = c69e4233b679437b8eb3b811680fa196-e30b47ad42124444aadeabc9924e8681
I20240502 14:15:22.296810 25509 tablet_copy_source_session.cc:205] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.302266 27248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30b47ad42124444aadeabc9924e8681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.318817 25489 consensus_queue.cc:237] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:22.271095 27232 tablet_copy_client.cc:783] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.319723 27232 tablet_copy_client.cc:647] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.323454 25359 raft_consensus.cc:1261] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
W20240502 14:15:22.324461 25422 consensus_peers.cc:485] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 232f776da8074d5bb2643f4add1cb06a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.325206 27033 consensus_queue.cc:1035] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240502 14:15:22.326097 27248 tablet_copy_client.cc:783] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.326403 27248 tablet_copy_client.cc:647] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.329085 27040 raft_consensus.cc:2932] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } } }
I20240502 14:15:22.333997 27243 log.cc:824] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:22.335469 27232 tablet_copy_client.cc:523] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:22.336472 25356 raft_consensus.cc:2932] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } } }
I20240502 14:15:22.337134 25080 catalog_manager.cc:5484] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index -1 to 61, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.349391 27268 ts_tablet_manager.cc:902] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:22.351750 27232 tablet_bootstrap.cc:492] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap starting.
I20240502 14:15:22.356961 27248 tablet_copy_client.cc:523] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:22.373019 27268 tablet_copy_client.cc:308] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:22.381496 27259 tablet_copy_client.cc:308] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:22.389297 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet 60e291dc59934bc38239bb84b76975c9 from peer 2bf9b3d134bc4ac79a05d942fc2fb68b ({username='slave'} at 127.21.58.69:52799)
I20240502 14:15:22.389480 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet 60e291dc59934bc38239bb84b76975c9 from peer 2bf9b3d134bc4ac79a05d942fc2fb68b at {username='slave'} at 127.21.58.69:52799: session id = 2bf9b3d134bc4ac79a05d942fc2fb68b-60e291dc59934bc38239bb84b76975c9
I20240502 14:15:22.392488 25509 tablet_copy_source_session.cc:205] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.396356 27248 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: Bootstrap starting.
I20240502 14:15:22.398046 27268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e291dc59934bc38239bb84b76975c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.408764 27268 tablet_copy_client.cc:783] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.409090 27268 tablet_copy_client.cc:647] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.414355 27268 tablet_copy_client.cc:523] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:22.421177 27232 log.cc:824] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:22.422871 27268 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap starting.
I20240502 14:15:22.442797 25489 consensus_queue.cc:237] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:22.451002 25359 raft_consensus.cc:1261] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 14:15:22.470209 27090 consensus_queue.cc:1035] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
W20240502 14:15:22.478804 25425 consensus_peers.cc:485] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755): Couldn't send request to peer 2bf9b3d134bc4ac79a05d942fc2fb68b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ead87caf4f9c4695a9c1b8b64852cabf. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.483692 27217 raft_consensus.cc:2932] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:22.491464 25356 raft_consensus.cc:2932] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:22.496609 27288 ts_tablet_manager.cc:902] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: Initiating tablet copy from peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:22.496749 25082 catalog_manager.cc:5484] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index -1 to 86, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.528661 27288 tablet_copy_client.cc:308] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46045
I20240502 14:15:22.544512 25248 tablet_copy_service.cc:140] P a5cb74a14a8d434e986ac7d988731e8a: Received BeginTabletCopySession request for tablet de14694a11e14f6082bba7fa70f51553 from peer c69e4233b679437b8eb3b811680fa196 ({username='slave'} at 127.21.58.68:34391)
I20240502 14:15:22.544795 25248 tablet_copy_service.cc:161] P a5cb74a14a8d434e986ac7d988731e8a: Beginning new tablet copy session on tablet de14694a11e14f6082bba7fa70f51553 from peer c69e4233b679437b8eb3b811680fa196 at {username='slave'} at 127.21.58.68:34391: session id = c69e4233b679437b8eb3b811680fa196-de14694a11e14f6082bba7fa70f51553
I20240502 14:15:22.552755 25248 tablet_copy_source_session.cc:205] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.555886 27288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de14694a11e14f6082bba7fa70f51553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.562060 27288 tablet_copy_client.cc:783] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.562300 27288 tablet_copy_client.cc:647] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.563324 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet 0723f00c11d744f8b244a3d003ce7b7a from peer 103df70e837147d9a992da0e46ece12e ({username='slave'} at 127.21.58.71:55941)
I20240502 14:15:22.563482 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet 0723f00c11d744f8b244a3d003ce7b7a from peer 103df70e837147d9a992da0e46ece12e at {username='slave'} at 127.21.58.71:55941: session id = 103df70e837147d9a992da0e46ece12e-0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:22.564427 27288 tablet_copy_client.cc:523] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:22.566038 25509 tablet_copy_source_session.cc:205] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.569229 27259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723f00c11d744f8b244a3d003ce7b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.578209 27288 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: Bootstrap starting.
I20240502 14:15:22.579735 25489 consensus_queue.cc:237] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:22.581172 27259 tablet_copy_client.cc:783] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.581457 27259 tablet_copy_client.cc:647] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.581656 25227 raft_consensus.cc:1261] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240502 14:15:22.582237 27045 consensus_queue.cc:1035] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 14:15:22.584436 27259 tablet_copy_client.cc:523] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:15:22.584841 25293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.65:46045, user_credentials={real_user=slave}} blocked reactor thread for 47471us
I20240502 14:15:22.584885 27217 raft_consensus.cc:2932] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index -1 to 85, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
W20240502 14:15:22.586858 25422 consensus_peers.cc:485] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70:35761): Couldn't send request to peer 6365836c0f2b428bbeb4d69b9ceda2fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcc673d4727a4c049228070d9e2ada05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.587944 25227 raft_consensus.cc:2932] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index -1 to 85, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
I20240502 14:15:22.591696 27288 log.cc:824] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:22.592492 27259 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: Bootstrap starting.
I20240502 14:15:22.595019 27298 ts_tablet_manager.cc:902] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: Initiating tablet copy from peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:22.601089 25082 catalog_manager.cc:5484] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index -1 to 85, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
I20240502 14:15:22.620780 27288 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:22.624344 27288 tablet_bootstrap.cc:492] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: Bootstrap complete.
I20240502 14:15:22.627564 27288 ts_tablet_manager.cc:1367] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.002s
I20240502 14:15:22.631846 27298 tablet_copy_client.cc:308] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46045
I20240502 14:15:22.640383 25248 tablet_copy_service.cc:140] P a5cb74a14a8d434e986ac7d988731e8a: Received BeginTabletCopySession request for tablet 503e0727a7a14863847f375032cb8144 from peer 6365836c0f2b428bbeb4d69b9ceda2fe ({username='slave'} at 127.21.58.70:42037)
I20240502 14:15:22.640568 25248 tablet_copy_service.cc:161] P a5cb74a14a8d434e986ac7d988731e8a: Beginning new tablet copy session on tablet 503e0727a7a14863847f375032cb8144 from peer 6365836c0f2b428bbeb4d69b9ceda2fe at {username='slave'} at 127.21.58.70:42037: session id = 6365836c0f2b428bbeb4d69b9ceda2fe-503e0727a7a14863847f375032cb8144
I20240502 14:15:22.641707 27243 tablet_bootstrap.cc:492] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=90 ignored=0} inserts{seen=721 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:22.642247 27243 tablet_bootstrap.cc:492] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap complete.
I20240502 14:15:22.642686 27243 ts_tablet_manager.cc:1367] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent bootstrapping tablet: real 0.394s user 0.159s sys 0.012s
I20240502 14:15:22.643247 25248 tablet_copy_source_session.cc:205] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.644462 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503e0727a7a14863847f375032cb8144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.649602 27243 raft_consensus.cc:348] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:22.650085 27243 raft_consensus.cc:729] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6365836c0f2b428bbeb4d69b9ceda2fe, State: Initialized, Role: LEARNER
I20240502 14:15:22.650656 27243 consensus_queue.cc:260] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:22.653425 27298 tablet_copy_client.cc:783] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.653767 27298 tablet_copy_client.cc:647] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.658768 27243 ts_tablet_manager.cc:1398] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20240502 14:15:22.658741 27298 tablet_copy_client.cc:523] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:22.662009 27298 tablet_bootstrap.cc:492] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap starting.
I20240502 14:15:22.663235 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-57558d7d2f9943dc8fe5fd127f3062e8 received from {username='slave'} at 127.21.58.70:60527
I20240502 14:15:22.663370 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-57558d7d2f9943dc8fe5fd127f3062e8 on tablet 57558d7d2f9943dc8fe5fd127f3062e8 with peer 6365836c0f2b428bbeb4d69b9ceda2fe
I20240502 14:15:22.668963 27288 raft_consensus.cc:348] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:22.669467 27288 raft_consensus.cc:729] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c69e4233b679437b8eb3b811680fa196, State: Initialized, Role: LEARNER
I20240502 14:15:22.670018 27288 consensus_queue.cc:260] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:22.712172 27288 ts_tablet_manager.cc:1398] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196: Time spent starting tablet: real 0.084s user 0.009s sys 0.003s
I20240502 14:15:22.744449 25248 tablet_copy_service.cc:342] P a5cb74a14a8d434e986ac7d988731e8a: Request end of tablet copy session c69e4233b679437b8eb3b811680fa196-de14694a11e14f6082bba7fa70f51553 received from {username='slave'} at 127.21.58.68:34391
I20240502 14:15:22.744594 25248 tablet_copy_service.cc:433] P a5cb74a14a8d434e986ac7d988731e8a: ending tablet copy session c69e4233b679437b8eb3b811680fa196-de14694a11e14f6082bba7fa70f51553 on tablet de14694a11e14f6082bba7fa70f51553 with peer c69e4233b679437b8eb3b811680fa196
I20240502 14:15:22.764482 25227 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
W20240502 14:15:22.766971 25164 consensus_peers.cc:485] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a -> Peer c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117): Couldn't send request to peer c69e4233b679437b8eb3b811680fa196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a68699dbcae34cfc94f2559e0516e5e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.778779 25359 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 14:15:22.781311 27151 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 14:15:22.789902 27151 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:22.798009 25082 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index -1 to 86, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.799373 25359 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:22.852474 27259 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=719 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:22.853040 27259 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: Bootstrap complete.
I20240502 14:15:22.854393 27259 ts_tablet_manager.cc:1367] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: Time spent bootstrapping tablet: real 0.262s user 0.132s sys 0.024s
I20240502 14:15:22.855849 27259 raft_consensus.cc:348] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:22.856175 27259 raft_consensus.cc:729] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 103df70e837147d9a992da0e46ece12e, State: Initialized, Role: LEARNER
I20240502 14:15:22.856427 27259 consensus_queue.cc:260] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:22.860716 26546 raft_consensus.cc:1203] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:22.864661 27259 ts_tablet_manager.cc:1398] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240502 14:15:22.866850 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 103df70e837147d9a992da0e46ece12e-0723f00c11d744f8b244a3d003ce7b7a received from {username='slave'} at 127.21.58.71:55941
I20240502 14:15:22.866990 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 103df70e837147d9a992da0e46ece12e-0723f00c11d744f8b244a3d003ce7b7a on tablet 0723f00c11d744f8b244a3d003ce7b7a with peer 103df70e837147d9a992da0e46ece12e
I20240502 14:15:22.884423 25359 consensus_queue.cc:237] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:22.893826 25489 raft_consensus.cc:1261] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 14:15:22.895776 27169 consensus_queue.cc:1035] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 14:15:22.898996 26440 raft_consensus.cc:2932] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } } }
I20240502 14:15:22.910488 25489 raft_consensus.cc:2932] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } } }
I20240502 14:15:22.920080 25080 catalog_manager.cc:5484] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index -1 to 75, NON_VOTER 103df70e837147d9a992da0e46ece12e (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.931886 27259 ts_tablet_manager.cc:902] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:22.973151 27328 ts_tablet_manager.cc:902] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:22.981916 26807 raft_consensus.cc:1203] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.73->[1.74-1.97] Dedup: 1.90->[1.91-1.97]
W20240502 14:15:22.997182 25292 consensus_peers.cc:485] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e -> Peer 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231): Couldn't send request to peer 103df70e837147d9a992da0e46ece12e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 291df3cd98de4152ab6aaf069bc45a36. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:23.005028 27232 tablet_bootstrap.cc:492] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=79 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:15:23.013932 27232 tablet_bootstrap.cc:492] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap complete.
I20240502 14:15:23.014448 27232 ts_tablet_manager.cc:1367] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent bootstrapping tablet: real 0.664s user 0.137s sys 0.020s
W20240502 14:15:23.023485 27268 log.cc:924] Time spent T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Append to log took a long time: real 0.058s user 0.002s sys 0.000s
I20240502 14:15:23.037856 27232 raft_consensus.cc:348] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:23.040761 25359 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
W20240502 14:15:23.043037 25295 consensus_peers.cc:485] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e -> Peer 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755): Couldn't send request to peer 2bf9b3d134bc4ac79a05d942fc2fb68b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 537b91a6ab124022a1cdfb077055d06f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:23.043797 25489 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 14:15:23.044278 26440 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 14:15:23.050002 27232 raft_consensus.cc:729] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2bf9b3d134bc4ac79a05d942fc2fb68b, State: Initialized, Role: LEARNER
I20240502 14:15:23.049760 26440 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:23.050603 27232 consensus_queue.cc:260] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:23.054219 25489 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:23.058180 25080 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index -1 to 74, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:23.059165 27232 ts_tablet_manager.cc:1398] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent starting tablet: real 0.045s user 0.012s sys 0.000s
I20240502 14:15:23.062981 25379 tablet_copy_service.cc:342] P c91321c0a57d4bec8eb3bfba60cee06e: Request end of tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-37ae6a3602fb4437a0e7c337494ea8e8 received from {username='slave'} at 127.21.58.69:35541
I20240502 14:15:23.063237 25379 tablet_copy_service.cc:433] P c91321c0a57d4bec8eb3bfba60cee06e: ending tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-37ae6a3602fb4437a0e7c337494ea8e8 on tablet 37ae6a3602fb4437a0e7c337494ea8e8 with peer 2bf9b3d134bc4ac79a05d942fc2fb68b
I20240502 14:15:23.104589 27298 tablet_bootstrap.cc:492] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=83 overwritten=0 applied=83 ignored=0} inserts{seen=768 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:23.118980 27298 tablet_bootstrap.cc:492] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap complete.
I20240502 14:15:23.119349 27298 ts_tablet_manager.cc:1367] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent bootstrapping tablet: real 0.457s user 0.165s sys 0.017s
I20240502 14:15:23.120271 27298 raft_consensus.cc:348] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:23.125530 27298 raft_consensus.cc:729] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6365836c0f2b428bbeb4d69b9ceda2fe, State: Initialized, Role: LEARNER
I20240502 14:15:23.130970 27259 tablet_copy_client.cc:308] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:23.131624 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet 232f776da8074d5bb2643f4add1cb06a from peer 103df70e837147d9a992da0e46ece12e ({username='slave'} at 127.21.58.71:55941)
I20240502 14:15:23.131768 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet 232f776da8074d5bb2643f4add1cb06a from peer 103df70e837147d9a992da0e46ece12e at {username='slave'} at 127.21.58.71:55941: session id = 103df70e837147d9a992da0e46ece12e-232f776da8074d5bb2643f4add1cb06a
I20240502 14:15:23.134935 25509 tablet_copy_source_session.cc:205] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:23.137519 27259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232f776da8074d5bb2643f4add1cb06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:23.139729 27298 consensus_queue.cc:260] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 83, Last appended: 1.83, Last appended by leader: 83, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:23.142823 27298 ts_tablet_manager.cc:1398] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:15:23.147905 25248 tablet_copy_service.cc:342] P a5cb74a14a8d434e986ac7d988731e8a: Request end of tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-503e0727a7a14863847f375032cb8144 received from {username='slave'} at 127.21.58.70:42037
I20240502 14:15:23.148020 25248 tablet_copy_service.cc:433] P a5cb74a14a8d434e986ac7d988731e8a: ending tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-503e0727a7a14863847f375032cb8144 on tablet 503e0727a7a14863847f375032cb8144 with peer 6365836c0f2b428bbeb4d69b9ceda2fe
I20240502 14:15:23.158958 27259 tablet_copy_client.cc:783] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:23.159197 27259 tablet_copy_client.cc:647] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:23.161490 27259 tablet_copy_client.cc:523] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:23.162937 27045 raft_consensus.cc:1050] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe to VOTER
I20240502 14:15:23.164250 27045 consensus_queue.cc:237] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:23.165925 27259 tablet_bootstrap.cc:492] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: Bootstrap starting.
I20240502 14:15:23.166951 25359 raft_consensus.cc:1261] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 14:15:23.166993 26807 raft_consensus.cc:1261] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 14:15:23.167575 27328 tablet_copy_client.cc:308] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:23.171497 27090 consensus_queue.cc:1035] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 14:15:23.171741 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet ead87caf4f9c4695a9c1b8b64852cabf from peer 2bf9b3d134bc4ac79a05d942fc2fb68b ({username='slave'} at 127.21.58.69:52799)
I20240502 14:15:23.171916 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet ead87caf4f9c4695a9c1b8b64852cabf from peer 2bf9b3d134bc4ac79a05d942fc2fb68b at {username='slave'} at 127.21.58.69:52799: session id = 2bf9b3d134bc4ac79a05d942fc2fb68b-ead87caf4f9c4695a9c1b8b64852cabf
I20240502 14:15:23.172027 27090 consensus_queue.cc:1035] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 99, Time since last communication: 0.000s
I20240502 14:15:23.175688 25509 tablet_copy_source_session.cc:205] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:23.182346 27328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead87caf4f9c4695a9c1b8b64852cabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:23.183212 27033 raft_consensus.cc:2932] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 74 to 102, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.188797 27328 tablet_copy_client.cc:783] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:23.189043 27328 tablet_copy_client.cc:647] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:23.202932 25359 raft_consensus.cc:2932] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 74 to 102, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.207736 25080 catalog_manager.cc:5484] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 74 to 102, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:23.210297 27248 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:15:23.210768 27248 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: Bootstrap complete.
I20240502 14:15:23.210139 26807 raft_consensus.cc:2932] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 74 to 102, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.211053 27248 ts_tablet_manager.cc:1367] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: Time spent bootstrapping tablet: real 0.815s user 0.130s sys 0.010s
I20240502 14:15:23.212020 27248 raft_consensus.cc:348] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:23.223328 27298 ts_tablet_manager.cc:902] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:23.223644 27328 tablet_copy_client.cc:523] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:23.226925 27328 tablet_bootstrap.cc:492] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap starting.
I20240502 14:15:23.235502 25489 consensus_queue.cc:237] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:23.250252 27298 tablet_copy_client.cc:308] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:23.251562 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet fcc673d4727a4c049228070d9e2ada05 from peer 6365836c0f2b428bbeb4d69b9ceda2fe ({username='slave'} at 127.21.58.70:60527)
I20240502 14:15:23.251737 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet fcc673d4727a4c049228070d9e2ada05 from peer 6365836c0f2b428bbeb4d69b9ceda2fe at {username='slave'} at 127.21.58.70:60527: session id = 6365836c0f2b428bbeb4d69b9ceda2fe-fcc673d4727a4c049228070d9e2ada05
I20240502 14:15:23.254642 25509 tablet_copy_source_session.cc:205] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:23.255841 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc673d4727a4c049228070d9e2ada05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:23.258276 26807 raft_consensus.cc:1261] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 14:15:23.264468 27298 tablet_copy_client.cc:783] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:23.265339 27298 tablet_copy_client.cc:647] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:23.269220 27248 raft_consensus.cc:729] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c69e4233b679437b8eb3b811680fa196, State: Initialized, Role: LEARNER
I20240502 14:15:23.269477 27248 consensus_queue.cc:260] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:23.271034 27248 ts_tablet_manager.cc:1398] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196: Time spent starting tablet: real 0.060s user 0.023s sys 0.004s
I20240502 14:15:23.271728 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session c69e4233b679437b8eb3b811680fa196-e30b47ad42124444aadeabc9924e8681 received from {username='slave'} at 127.21.58.68:40165
I20240502 14:15:23.271878 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session c69e4233b679437b8eb3b811680fa196-e30b47ad42124444aadeabc9924e8681 on tablet e30b47ad42124444aadeabc9924e8681 with peer c69e4233b679437b8eb3b811680fa196
I20240502 14:15:23.275080 27298 tablet_copy_client.cc:523] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:23.282598 27248 ts_tablet_manager.cc:902] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: Initiating tablet copy from peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:23.284144 27298 tablet_bootstrap.cc:492] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap starting.
I20240502 14:15:23.294814 27033 consensus_queue.cc:1035] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 14:15:23.316898 27090 raft_consensus.cc:2932] T 57558d7d2f9943dc8fe5fd127f3062e8 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.318250 27248 tablet_copy_client.cc:308] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46045
I20240502 14:15:23.319450 25248 tablet_copy_service.cc:140] P a5cb74a14a8d434e986ac7d988731e8a: Received BeginTabletCopySession request for tablet a68699dbcae34cfc94f2559e0516e5e8 from peer c69e4233b679437b8eb3b811680fa196 ({username='slave'} at 127.21.58.68:34391)
I20240502 14:15:23.319593 25248 tablet_copy_service.cc:161] P a5cb74a14a8d434e986ac7d988731e8a: Beginning new tablet copy session on tablet a68699dbcae34cfc94f2559e0516e5e8 from peer c69e4233b679437b8eb3b811680fa196 at {username='slave'} at 127.21.58.68:34391: session id = c69e4233b679437b8eb3b811680fa196-a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:23.322254 25248 tablet_copy_source_session.cc:205] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:23.324478 26940 raft_consensus.cc:1203] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.70->[1.71-1.94] Dedup: 1.88->[1.89-1.94]
I20240502 14:15:23.324319 26807 raft_consensus.cc:2932] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.327785 27248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68699dbcae34cfc94f2559e0516e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:23.329102 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57558d7d2f9943dc8fe5fd127f3062e8 with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:23.334359 27248 tablet_copy_client.cc:783] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:23.334594 27248 tablet_copy_client.cc:647] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:23.334329 25082 catalog_manager.cc:5484] T 57558d7d2f9943dc8fe5fd127f3062e8 P 6365836c0f2b428bbeb4d69b9ceda2fe reported cstate change: config changed from index 102 to 103, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.338562 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet 57558d7d2f9943dc8fe5fd127f3062e8 with delete_type TABLET_DATA_TOMBSTONED (TS c91321c0a57d4bec8eb3bfba60cee06e not found in new config with opid_index 103) from {username='slave'} at 127.0.0.1:43744
I20240502 14:15:23.340723 27370 tablet_replica.cc:330] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: stopping tablet replica
I20240502 14:15:23.342711 27370 raft_consensus.cc:2229] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:23.344066 27248 tablet_copy_client.cc:523] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:23.345045 27259 tablet_bootstrap.cc:492] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: Bootstrap replayed 1/1 log segments. Stats: ops{read=67 overwritten=0 applied=67 ignored=0} inserts{seen=698 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:23.346915 27370 raft_consensus.cc:2258] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:23.348484 27248 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: Bootstrap starting.
I20240502 14:15:23.349982 27259 tablet_bootstrap.cc:492] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: Bootstrap complete.
I20240502 14:15:23.357731 27259 ts_tablet_manager.cc:1367] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: Time spent bootstrapping tablet: real 0.192s user 0.115s sys 0.014s
I20240502 14:15:23.362445 27370 ts_tablet_manager.cc:1875] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:23.365684 27259 raft_consensus.cc:348] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:23.366693 27259 raft_consensus.cc:729] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 103df70e837147d9a992da0e46ece12e, State: Initialized, Role: LEARNER
I20240502 14:15:23.366973 27259 consensus_queue.cc:260] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 67, Last appended: 1.67, Last appended by leader: 67, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:23.370036 27259 ts_tablet_manager.cc:1398] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:15:23.378846 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 103df70e837147d9a992da0e46ece12e-232f776da8074d5bb2643f4add1cb06a received from {username='slave'} at 127.21.58.71:55941
I20240502 14:15:23.379032 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 103df70e837147d9a992da0e46ece12e-232f776da8074d5bb2643f4add1cb06a on tablet 232f776da8074d5bb2643f4add1cb06a with peer 103df70e837147d9a992da0e46ece12e
I20240502 14:15:23.380226 27217 raft_consensus.cc:1050] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 103df70e837147d9a992da0e46ece12e to VOTER
I20240502 14:15:23.380923 27217 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } }
I20240502 14:15:23.384120 26940 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240502 14:15:23.384672 27370 ts_tablet_manager.cc:1888] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.102
I20240502 14:15:23.384871 27370 log.cc:1196] T 57558d7d2f9943dc8fe5fd127f3062e8 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/57558d7d2f9943dc8fe5fd127f3062e8
I20240502 14:15:23.385965 25228 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240502 14:15:23.386514 27205 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 14:15:23.387553 25071 catalog_manager.cc:4830] TS c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): tablet 57558d7d2f9943dc8fe5fd127f3062e8 (table rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]) successfully deleted
I20240502 14:15:23.390620 27205 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 71 to 97, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.393889 26940 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 71 to 97, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.400852 25082 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e reported cstate change: config changed from index 71 to 97, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.409749 26940 raft_consensus.cc:1203] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.60->[1.61-1.72] Dedup: 1.67->[1.68-1.72]
I20240502 14:15:23.429863 25489 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } }
W20240502 14:15:23.430616 27090 consensus_queue.cc:1159] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a5cb74a14a8d434e986ac7d988731e8a" responder_term: 1 status { last_received { term: 1 index: 96 } last_committed_idx: 95 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)" } } last_received_current_leader { term: 1 index: 96 } }
I20240502 14:15:23.441581 26941 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240502 14:15:23.454869 27033 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240502 14:15:23.463263 27379 raft_consensus.cc:1050] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: attempting to promote NON_VOTER c69e4233b679437b8eb3b811680fa196 to VOTER
I20240502 14:15:23.464730 27379 consensus_queue.cc:237] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:23.473345 26677 raft_consensus.cc:1203] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.100] Dedup: 1.80->[1.81-1.100]
I20240502 14:15:23.479193 27090 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.482893 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0723f00c11d744f8b244a3d003ce7b7a with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:23.484555 27045 raft_consensus.cc:1050] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 103df70e837147d9a992da0e46ece12e to VOTER
I20240502 14:15:23.487027 27045 consensus_queue.cc:237] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } }
I20240502 14:15:23.489256 27259 ts_tablet_manager.cc:902] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: Initiating tablet copy from peer c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:23.493656 25357 raft_consensus.cc:1261] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 14:15:23.493731 26940 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.494283 26939 raft_consensus.cc:1261] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 14:15:23.496632 27090 consensus_queue.cc:1035] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 14:15:23.501983 25082 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e reported cstate change: config changed from index 97 to 98, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.504913 27090 consensus_queue.cc:1035] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 14:15:23.505975 25207 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723f00c11d744f8b244a3d003ce7b7a with delete_type TABLET_DATA_TOMBSTONED (TS a5cb74a14a8d434e986ac7d988731e8a not found in new config with opid_index 98) from {username='slave'} at 127.0.0.1:34208
I20240502 14:15:23.522982 26941 raft_consensus.cc:1261] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:23.523604 27151 consensus_queue.cc:1035] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:23.523938 27387 tablet_replica.cc:330] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: stopping tablet replica
I20240502 14:15:23.526242 27387 raft_consensus.cc:2229] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:23.526614 27387 pending_rounds.cc:62] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Trying to abort 1 pending ops.
I20240502 14:15:23.527357 27387 pending_rounds.cc:68] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Aborting op as it isn't in flight: id { term: 1 index: 96 } timestamp: 7023244588466126848 op_type: WRITE_OP write_request { tablet_id: "0723f00c11d744f8b244a3d003ce7b7a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000ie\2465l\331s\221\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d7f2ebc91aabcacf" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023244588386963456 authz_token { token_data: "\010\242\302\316\261\006\"3\n\005slave\022*\n c27f89dbb8084518b0a53aa80d1ebe66\020\001\030\001 \001(\001" signature: "<redacted>""P\370\036\362\327\032SKj\310\304\271\213\243~\000\2151S\000^\202.\226G\267\211\244\267\214(\275\210\007B`/\232\314>h\346h{\343\350\231\003\247\373TaT\"8R\357\235\313\257\360\005\010k" signing_key_seq_num: 0 } } request_id { client_id: "aa610212a61148e5a19eff512ed3a3e1" seq_no: 276 first_incomplete_seq_no: 276 attempt_no: 0 }
I20240502 14:15:23.529073 27387 raft_consensus.cc:2258] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:23.529728 26807 raft_consensus.cc:1203] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.75->[1.76-1.95] Dedup: 1.83->[1.84-1.95]
I20240502 14:15:23.532068 26546 raft_consensus.cc:1261] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:23.532649 27379 consensus_queue.cc:1035] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:23.537026 27045 raft_consensus.cc:2932] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 61 to 75, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.547729 25359 raft_consensus.cc:2932] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 61 to 75, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.547889 25080 catalog_manager.cc:5484] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 61 to 75, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:23.559322 27387 ts_tablet_manager.cc:1875] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:23.568567 27379 raft_consensus.cc:1050] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a: attempting to promote NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe to VOTER
I20240502 14:15:23.569412 27379 consensus_queue.cc:237] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:23.569614 27259 tablet_copy_client.cc:308] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44323
I20240502 14:15:23.571993 26806 raft_consensus.cc:1261] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240502 14:15:23.571827 27057 raft_consensus.cc:2932] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:23.572671 27379 consensus_queue.cc:1035] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 14:15:23.576393 27151 raft_consensus.cc:2932] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 76 to 97, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.576498 26546 raft_consensus.cc:2932] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:23.577561 25080 catalog_manager.cc:5484] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 2 to 3, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:23.579568 26806 raft_consensus.cc:2932] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 76 to 97, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.590454 25080 catalog_manager.cc:5484] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe reported cstate change: config changed from index 76 to 97, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.594833 25228 consensus_queue.cc:237] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:23.602813 26806 raft_consensus.cc:1261] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240502 14:15:23.603289 27057 consensus_queue.cc:1035] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240502 14:15:23.605368 27151 raft_consensus.cc:2932] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.609059 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 503e0727a7a14863847f375032cb8144 with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 14:15:23.609757 25422 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.71:37231, user_credentials={real_user=slave}} blocked reactor thread for 58984.3us
I20240502 14:15:23.609652 26806 raft_consensus.cc:2932] T 503e0727a7a14863847f375032cb8144 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:23.610267 25080 catalog_manager.cc:5484] T 503e0727a7a14863847f375032cb8144 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 97 to 98, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:23.611351 26939 raft_consensus.cc:2932] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 61 to 75, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.615583 25488 raft_consensus.cc:1261] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240502 14:15:23.619038 25469 tablet_service.cc:1514] Processing DeleteTablet for tablet 503e0727a7a14863847f375032cb8144 with delete_type TABLET_DATA_TOMBSTONED (TS 1360ad293a3e4ecebf03dbf14cca9cb4 not found in new config with opid_index 98) from {username='slave'} at 127.0.0.1:40350
I20240502 14:15:23.636024 25489 consensus_queue.cc:237] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } }
I20240502 14:15:23.646847 26939 raft_consensus.cc:2932] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:23.651289 26941 raft_consensus.cc:1261] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240502 14:15:23.668732 26546 raft_consensus.cc:1203] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.56->[1.57-1.84] Dedup: 1.70->[1.71-1.84]
I20240502 14:15:23.671880 27405 tablet_replica.cc:330] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: stopping tablet replica
I20240502 14:15:23.672394 27405 raft_consensus.cc:2229] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:23.676366 27405 pending_rounds.cc:62] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Trying to abort 1 pending ops.
I20240502 14:15:23.677449 27387 ts_tablet_manager.cc:1888] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.96
I20240502 14:15:23.677636 27405 pending_rounds.cc:68] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Aborting op as it isn't in flight: id { term: 1 index: 96 } timestamp: 7023244589322747904 op_type: WRITE_OP write_request { tablet_id: "503e0727a7a14863847f375032cb8144" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\025\024\007;KW\210\315\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4a02fd15146ae5ab" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023244588997013504 authz_token { token_data: "\010\242\302\316\261\006\"3\n\005slave\022*\n c27f89dbb8084518b0a53aa80d1ebe66\020\001\030\001 \001(\001" signature: "<redacted>""P\370\036\362\327\032SKj\310\304\271\213\243~\000\2151S\000^\202.\226G\267\211\244\267\214(\275\210\007B`/\232\314>h\346h{\343\350\231\003\247\373TaT\"8R\357\235\313\257\360\005\010k" signing_key_seq_num: 0 } } request_id { client_id: "aa610212a61148e5a19eff512ed3a3e1" seq_no: 281 first_incomplete_seq_no: 281 attempt_no: 0 }
I20240502 14:15:23.688369 25379 tablet_copy_service.cc:140] P c91321c0a57d4bec8eb3bfba60cee06e: Received BeginTabletCopySession request for tablet 291df3cd98de4152ab6aaf069bc45a36 from peer 103df70e837147d9a992da0e46ece12e ({username='slave'} at 127.21.58.71:45169)
I20240502 14:15:23.688556 25379 tablet_copy_service.cc:161] P c91321c0a57d4bec8eb3bfba60cee06e: Beginning new tablet copy session on tablet 291df3cd98de4152ab6aaf069bc45a36 from peer 103df70e837147d9a992da0e46ece12e at {username='slave'} at 127.21.58.71:45169: session id = 103df70e837147d9a992da0e46ece12e-291df3cd98de4152ab6aaf069bc45a36
I20240502 14:15:23.692752 25379 tablet_copy_source_session.cc:205] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:23.693924 27259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291df3cd98de4152ab6aaf069bc45a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:23.698493 27408 ts_tablet_manager.cc:902] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: Initiating tablet copy from peer c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:23.708525 27268 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=99 ignored=0} inserts{seen=784 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:15:23.709043 27268 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap complete.
I20240502 14:15:23.709329 27268 ts_tablet_manager.cc:1367] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent bootstrapping tablet: real 1.287s user 0.151s sys 0.013s
I20240502 14:15:23.710275 27268 raft_consensus.cc:348] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:23.711144 27268 raft_consensus.cc:729] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2bf9b3d134bc4ac79a05d942fc2fb68b, State: Initialized, Role: LEARNER
I20240502 14:15:23.711365 27268 consensus_queue.cc:260] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:23.712374 27268 ts_tablet_manager.cc:1398] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 14:15:23.728883 27259 tablet_copy_client.cc:783] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:23.729211 27259 tablet_copy_client.cc:647] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:23.729960 27387 log.cc:1196] T 0723f00c11d744f8b244a3d003ce7b7a P a5cb74a14a8d434e986ac7d988731e8a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:23.730008 27090 consensus_queue.cc:1035] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 76, Time since last communication: 0.047s
I20240502 14:15:23.730094 27405 raft_consensus.cc:2258] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:23.736433 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-60e291dc59934bc38239bb84b76975c9 received from {username='slave'} at 127.21.58.69:52799
I20240502 14:15:23.737222 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-60e291dc59934bc38239bb84b76975c9 on tablet 60e291dc59934bc38239bb84b76975c9 with peer 2bf9b3d134bc4ac79a05d942fc2fb68b
I20240502 14:15:23.748869 27408 tablet_copy_client.cc:308] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44323
I20240502 14:15:23.750171 25379 tablet_copy_service.cc:140] P c91321c0a57d4bec8eb3bfba60cee06e: Received BeginTabletCopySession request for tablet 537b91a6ab124022a1cdfb077055d06f from peer 2bf9b3d134bc4ac79a05d942fc2fb68b ({username='slave'} at 127.21.58.69:35541)
I20240502 14:15:23.750330 25379 tablet_copy_service.cc:161] P c91321c0a57d4bec8eb3bfba60cee06e: Beginning new tablet copy session on tablet 537b91a6ab124022a1cdfb077055d06f from peer 2bf9b3d134bc4ac79a05d942fc2fb68b at {username='slave'} at 127.21.58.69:35541: session id = 2bf9b3d134bc4ac79a05d942fc2fb68b-537b91a6ab124022a1cdfb077055d06f
I20240502 14:15:23.753046 25379 tablet_copy_source_session.cc:205] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:23.756356 27408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b91a6ab124022a1cdfb077055d06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:23.757128 27103 raft_consensus.cc:1050] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e: attempting to promote NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b to VOTER
I20240502 14:15:23.757983 27103 consensus_queue.cc:237] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:23.759805 25227 raft_consensus.cc:1261] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 14:15:23.760339 27168 consensus_queue.cc:1035] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 14:15:23.764046 27328 tablet_bootstrap.cc:492] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=778 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:23.764431 27328 tablet_bootstrap.cc:492] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap complete.
I20240502 14:15:23.764695 27328 ts_tablet_manager.cc:1367] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent bootstrapping tablet: real 0.538s user 0.162s sys 0.001s
I20240502 14:15:23.764873 26440 raft_consensus.cc:2932] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 62 to 102, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:23.765396 27328 raft_consensus.cc:348] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:23.765599 27328 raft_consensus.cc:729] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2bf9b3d134bc4ac79a05d942fc2fb68b, State: Initialized, Role: LEARNER
I20240502 14:15:23.765796 27328 consensus_queue.cc:260] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:23.766384 27328 ts_tablet_manager.cc:1398] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:23.767966 25224 raft_consensus.cc:1203] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.100->[1.101-1.101] Dedup: 1.101->[]
I20240502 14:15:23.768700 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-ead87caf4f9c4695a9c1b8b64852cabf received from {username='slave'} at 127.21.58.69:52799
I20240502 14:15:23.768847 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-ead87caf4f9c4695a9c1b8b64852cabf on tablet ead87caf4f9c4695a9c1b8b64852cabf with peer 2bf9b3d134bc4ac79a05d942fc2fb68b
I20240502 14:15:23.770128 25227 raft_consensus.cc:2932] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 62 to 102, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:23.773552 26677 raft_consensus.cc:1261] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 14:15:23.777494 27168 consensus_queue.cc:1035] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 14:15:23.780881 26677 raft_consensus.cc:2932] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 62 to 102, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:23.782665 25072 catalog_manager.cc:4830] TS a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): tablet 0723f00c11d744f8b244a3d003ce7b7a (table rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]) successfully deleted
I20240502 14:15:23.786620 25080 catalog_manager.cc:5484] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 62 to 102, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:23.786623 27405 ts_tablet_manager.cc:1875] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:23.796859 27408 tablet_copy_client.cc:783] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:23.797111 27408 tablet_copy_client.cc:647] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:23.796952 25356 consensus_queue.cc:237] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:23.800561 26677 raft_consensus.cc:1261] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 14:15:23.802829 27408 tablet_copy_client.cc:523] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:23.809744 27168 consensus_queue.cc:1035] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 14:15:23.814244 27169 raft_consensus.cc:2932] T 37ae6a3602fb4437a0e7c337494ea8e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:23.814522 26677 raft_consensus.cc:2932] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:23.819953 25080 catalog_manager.cc:5484] T 37ae6a3602fb4437a0e7c337494ea8e8 P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 102 to 103, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:23.820703 25071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37ae6a3602fb4437a0e7c337494ea8e8 with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:23.824950 27405 ts_tablet_manager.cc:1888] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.96
I20240502 14:15:23.825565 25207 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ae6a3602fb4437a0e7c337494ea8e8 with delete_type TABLET_DATA_TOMBSTONED (TS a5cb74a14a8d434e986ac7d988731e8a not found in new config with opid_index 103) from {username='slave'} at 127.0.0.1:34208
I20240502 14:15:23.828229 27405 log.cc:1196] T 503e0727a7a14863847f375032cb8144 P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/503e0727a7a14863847f375032cb8144
I20240502 14:15:23.832468 25072 catalog_manager.cc:4830] TS 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255): tablet 503e0727a7a14863847f375032cb8144 (table rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]) successfully deleted
I20240502 14:15:23.832973 27090 raft_consensus.cc:1050] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER c69e4233b679437b8eb3b811680fa196 to VOTER
I20240502 14:15:23.833096 27248 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=90 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:23.833269 27408 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap starting.
I20240502 14:15:23.833544 27248 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: Bootstrap complete.
I20240502 14:15:23.833607 27090 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:23.833812 27248 ts_tablet_manager.cc:1367] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: Time spent bootstrapping tablet: real 0.486s user 0.136s sys 0.009s
I20240502 14:15:23.834923 27248 raft_consensus.cc:348] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:23.835299 26546 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 14:15:23.835609 25224 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 14:15:23.835942 27033 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 14:15:23.837294 27033 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 14:15:23.835191 27248 raft_consensus.cc:729] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c69e4233b679437b8eb3b811680fa196, State: Initialized, Role: LEARNER
I20240502 14:15:23.837950 27248 consensus_queue.cc:260] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:23.838529 27248 ts_tablet_manager.cc:1398] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:15:23.838934 27033 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index 57 to 86, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:23.839691 25248 tablet_copy_service.cc:342] P a5cb74a14a8d434e986ac7d988731e8a: Request end of tablet copy session c69e4233b679437b8eb3b811680fa196-a68699dbcae34cfc94f2559e0516e5e8 received from {username='slave'} at 127.21.58.68:34391
I20240502 14:15:23.839833 25248 tablet_copy_service.cc:433] P a5cb74a14a8d434e986ac7d988731e8a: ending tablet copy session c69e4233b679437b8eb3b811680fa196-a68699dbcae34cfc94f2559e0516e5e8 on tablet a68699dbcae34cfc94f2559e0516e5e8 with peer c69e4233b679437b8eb3b811680fa196
I20240502 14:15:23.841277 26545 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 57 to 86, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:23.843597 25226 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 57 to 86, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:23.849756 25079 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 57 to 86, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:23.854220 27387 tablet_replica.cc:330] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: stopping tablet replica
I20240502 14:15:23.854415 27387 raft_consensus.cc:2229] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:23.855430 27387 raft_consensus.cc:2258] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:23.856616 27387 ts_tablet_manager.cc:1875] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:23.858508 27045 raft_consensus.cc:2932] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index 75 to 78, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.863224 27387 ts_tablet_manager.cc:1888] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.102
I20240502 14:15:23.863168 25082 catalog_manager.cc:5484] T 232f776da8074d5bb2643f4add1cb06a P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 75 to 78, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:23.863365 27387 log.cc:1196] T 37ae6a3602fb4437a0e7c337494ea8e8 P a5cb74a14a8d434e986ac7d988731e8a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/37ae6a3602fb4437a0e7c337494ea8e8
I20240502 14:15:23.863896 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 232f776da8074d5bb2643f4add1cb06a with cas_config_opid_index 75: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:23.864266 25072 catalog_manager.cc:4830] TS a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): tablet 37ae6a3602fb4437a0e7c337494ea8e8 (table rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]) successfully deleted
I20240502 14:15:23.868065 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet 232f776da8074d5bb2643f4add1cb06a with delete_type TABLET_DATA_TOMBSTONED (TS c91321c0a57d4bec8eb3bfba60cee06e not found in new config with opid_index 78) from {username='slave'} at 127.0.0.1:43744
I20240502 14:15:23.868399 27370 tablet_replica.cc:330] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: stopping tablet replica
I20240502 14:15:23.868585 27370 raft_consensus.cc:2229] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:23.869120 27370 raft_consensus.cc:2258] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:23.870807 26676 raft_consensus.cc:1203] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.87->[1.88-1.116] Dedup: 1.100->[1.101-1.116]
I20240502 14:15:23.876035 27370 ts_tablet_manager.cc:1875] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:23.884972 27370 ts_tablet_manager.cc:1888] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.76
I20240502 14:15:23.868829 26939 raft_consensus.cc:2932] T 232f776da8074d5bb2643f4add1cb06a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 75 to 78, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:23.970423 27370 log.cc:1196] T 232f776da8074d5bb2643f4add1cb06a P c91321c0a57d4bec8eb3bfba60cee06e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/232f776da8074d5bb2643f4add1cb06a
I20240502 14:15:23.971235 25071 catalog_manager.cc:4830] TS c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): tablet 232f776da8074d5bb2643f4add1cb06a (table rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]) successfully deleted
I20240502 14:15:23.982705 26677 raft_consensus.cc:1203] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.85->[1.86-1.102] Dedup: 1.93->[1.94-1.102]
I20240502 14:15:24.004557 27259 tablet_copy_client.cc:523] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.033326 27259 tablet_bootstrap.cc:492] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: Bootstrap starting.
I20240502 14:15:24.063625 27205 raft_consensus.cc:1050] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b to VOTER
I20240502 14:15:24.064297 27205 consensus_queue.cc:237] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:24.066011 26676 raft_consensus.cc:1261] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 14:15:24.066406 25359 raft_consensus.cc:1261] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 14:15:24.066491 27090 consensus_queue.cc:1035] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 14:15:24.067377 27090 consensus_queue.cc:1035] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 14:15:24.068903 27090 raft_consensus.cc:2932] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 86 to 104, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.072818 25356 raft_consensus.cc:2932] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 86 to 104, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.074468 26546 raft_consensus.cc:1203] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.85->[1.86-1.96] Dedup: 1.90->[1.91-1.96]
I20240502 14:15:24.077919 25082 catalog_manager.cc:5484] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index 86 to 104, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.079502 26676 raft_consensus.cc:2932] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 86 to 104, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.099138 27033 raft_consensus.cc:1050] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b to VOTER
I20240502 14:15:24.099865 27033 consensus_queue.cc:237] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:24.102360 25359 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240502 14:15:24.104218 27298 tablet_bootstrap.cc:492] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.104226 27217 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240502 14:15:24.104643 27298 tablet_bootstrap.cc:492] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap complete.
I20240502 14:15:24.104899 27298 ts_tablet_manager.cc:1367] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent bootstrapping tablet: real 0.821s user 0.141s sys 0.008s
I20240502 14:15:24.105715 27298 raft_consensus.cc:348] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:24.106177 27298 raft_consensus.cc:729] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6365836c0f2b428bbeb4d69b9ceda2fe, State: Initialized, Role: LEARNER
I20240502 14:15:24.106354 26677 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240502 14:15:24.106539 27298 consensus_queue.cc:260] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:24.107194 27298 ts_tablet_manager.cc:1398] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:24.107601 27040 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240502 14:15:24.107836 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-fcc673d4727a4c049228070d9e2ada05 received from {username='slave'} at 127.21.58.70:60527
I20240502 14:15:24.108019 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-fcc673d4727a4c049228070d9e2ada05 on tablet fcc673d4727a4c049228070d9e2ada05 with peer 6365836c0f2b428bbeb4d69b9ceda2fe
I20240502 14:15:24.110186 27217 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index 88 to 118, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.110834 26675 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 88 to 118, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.117514 25359 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 88 to 118, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.118466 25080 catalog_manager.cc:5484] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 88 to 118, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.131203 25489 consensus_queue.cc:237] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 1.119, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:24.134776 26676 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240502 14:15:24.135578 27045 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 118, Time since last communication: 0.000s
I20240502 14:15:24.141070 27205 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.120: config changed from index 118 to 120, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.142011 26677 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 118 to 120, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.149442 25082 catalog_manager.cc:5484] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 118 to 120, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:24.150803 27379 raft_consensus.cc:1050] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: attempting to promote NON_VOTER c69e4233b679437b8eb3b811680fa196 to VOTER
I20240502 14:15:24.151466 27379 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:24.154428 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60e291dc59934bc38239bb84b76975c9 with cas_config_opid_index 118: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:24.154651 26546 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240502 14:15:24.155364 27057 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240502 14:15:24.155634 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e291dc59934bc38239bb84b76975c9 with delete_type TABLET_DATA_TOMBSTONED (TS c91321c0a57d4bec8eb3bfba60cee06e not found in new config with opid_index 120) from {username='slave'} at 127.0.0.1:43744
I20240502 14:15:24.156347 27370 tablet_replica.cc:330] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: stopping tablet replica
I20240502 14:15:24.156556 27370 raft_consensus.cc:2229] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.156800 27370 raft_consensus.cc:2258] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.157720 27370 ts_tablet_manager.cc:1875] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.157578 27379 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 86 to 99, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:24.158780 26546 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 86 to 99, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:24.178427 25082 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 86 to 99, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:24.184639 27370 ts_tablet_manager.cc:1888] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.119
I20240502 14:15:24.184760 27370 log.cc:1196] T 60e291dc59934bc38239bb84b76975c9 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/60e291dc59934bc38239bb84b76975c9
I20240502 14:15:24.185402 25071 catalog_manager.cc:4830] TS c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): tablet 60e291dc59934bc38239bb84b76975c9 (table rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]) successfully deleted
I20240502 14:15:24.191423 25357 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240502 14:15:24.193169 27057 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240502 14:15:24.203011 25226 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.100, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:24.204362 26546 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240502 14:15:24.204823 27151 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 14:15:24.207089 27379 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.101: config changed from index 99 to 101, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:24.211030 26546 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index 99 to 101, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:24.212905 25082 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 99 to 101, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:24.218075 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a68699dbcae34cfc94f2559e0516e5e8 with cas_config_opid_index 99: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:24.225155 25356 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 86 to 99, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:24.233942 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet a68699dbcae34cfc94f2559e0516e5e8 with delete_type TABLET_DATA_TOMBSTONED (TS c91321c0a57d4bec8eb3bfba60cee06e not found in new config with opid_index 101) from {username='slave'} at 127.0.0.1:43744
I20240502 14:15:24.238428 27370 tablet_replica.cc:330] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: stopping tablet replica
I20240502 14:15:24.246351 27370 raft_consensus.cc:2229] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.253965 27370 raft_consensus.cc:2258] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.254982 27370 ts_tablet_manager.cc:1875] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.264286 26806 raft_consensus.cc:1203] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.84->[1.85-1.111] Dedup: 1.97->[1.98-1.111]
I20240502 14:15:24.269439 27370 ts_tablet_manager.cc:1888] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.100
I20240502 14:15:24.269551 27370 log.cc:1196] T a68699dbcae34cfc94f2559e0516e5e8 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:24.270306 25071 catalog_manager.cc:4830] TS c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): tablet a68699dbcae34cfc94f2559e0516e5e8 (table rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]) successfully deleted
I20240502 14:15:24.456756 27217 raft_consensus.cc:1050] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe to VOTER
I20240502 14:15:24.457420 27217 consensus_queue.cc:237] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.118, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:24.459318 25228 raft_consensus.cc:1261] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 14:15:24.464226 26806 raft_consensus.cc:1261] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 14:15:24.465009 27205 consensus_queue.cc:1035] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240502 14:15:24.466255 27205 consensus_queue.cc:1035] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240502 14:15:24.469372 27033 raft_consensus.cc:2932] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.119: config changed from index 85 to 119, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:24.473989 25228 raft_consensus.cc:2932] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 85 to 119, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:24.474308 26805 raft_consensus.cc:2932] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 85 to 119, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:24.479110 25082 catalog_manager.cc:5484] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 85 to 119, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:24.488224 25488 consensus_queue.cc:237] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 1.119, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:24.491309 26807 raft_consensus.cc:1261] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240502 14:15:24.492115 27045 consensus_queue.cc:1035] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20240502 14:15:24.496423 27040 raft_consensus.cc:2932] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.120: config changed from index 119 to 120, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:24.500738 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fcc673d4727a4c049228070d9e2ada05 with cas_config_opid_index 119: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:24.502646 25079 catalog_manager.cc:5484] T fcc673d4727a4c049228070d9e2ada05 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 119 to 120, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:24.504719 26807 raft_consensus.cc:2932] T fcc673d4727a4c049228070d9e2ada05 P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 119 to 120, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:24.507212 25207 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc673d4727a4c049228070d9e2ada05 with delete_type TABLET_DATA_TOMBSTONED (TS a5cb74a14a8d434e986ac7d988731e8a not found in new config with opid_index 120) from {username='slave'} at 127.0.0.1:34208
I20240502 14:15:24.528362 27439 tablet_replica.cc:330] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: stopping tablet replica
I20240502 14:15:24.528693 27439 raft_consensus.cc:2229] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.530375 27439 raft_consensus.cc:2258] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.535598 27439 ts_tablet_manager.cc:1875] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.548017 27259 tablet_bootstrap.cc:492] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: Bootstrap replayed 1/1 log segments. Stats: ops{read=82 overwritten=0 applied=82 ignored=0} inserts{seen=713 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.548498 27259 tablet_bootstrap.cc:492] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: Bootstrap complete.
I20240502 14:15:24.548789 27259 ts_tablet_manager.cc:1367] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: Time spent bootstrapping tablet: real 0.516s user 0.163s sys 0.060s
I20240502 14:15:24.549561 27259 raft_consensus.cc:348] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:24.549831 27259 raft_consensus.cc:729] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 103df70e837147d9a992da0e46ece12e, State: Initialized, Role: LEARNER
I20240502 14:15:24.550030 27259 consensus_queue.cc:260] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.82, Last appended by leader: 82, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: true } }
I20240502 14:15:24.552930 27439 ts_tablet_manager.cc:1888] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.119
I20240502 14:15:24.557040 27259 ts_tablet_manager.cc:1398] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 14:15:24.561878 27439 log.cc:1196] T fcc673d4727a4c049228070d9e2ada05 P a5cb74a14a8d434e986ac7d988731e8a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/fcc673d4727a4c049228070d9e2ada05
I20240502 14:15:24.563786 25072 catalog_manager.cc:4830] TS a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): tablet fcc673d4727a4c049228070d9e2ada05 (table rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]) successfully deleted
I20240502 14:15:24.573776 25379 tablet_copy_service.cc:342] P c91321c0a57d4bec8eb3bfba60cee06e: Request end of tablet copy session 103df70e837147d9a992da0e46ece12e-291df3cd98de4152ab6aaf069bc45a36 received from {username='slave'} at 127.21.58.71:45169
I20240502 14:15:24.573915 25379 tablet_copy_service.cc:433] P c91321c0a57d4bec8eb3bfba60cee06e: ending tablet copy session 103df70e837147d9a992da0e46ece12e-291df3cd98de4152ab6aaf069bc45a36 on tablet 291df3cd98de4152ab6aaf069bc45a36 with peer 103df70e837147d9a992da0e46ece12e
I20240502 14:15:24.654476 25488 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ead87caf4f9c4695a9c1b8b64852cabf"
dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54118
I20240502 14:15:24.654798 25488 raft_consensus.cc:595] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:24.682163 27408 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.690431 27408 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap complete.
I20240502 14:15:24.690802 27408 ts_tablet_manager.cc:1367] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent bootstrapping tablet: real 0.858s user 0.123s sys 0.023s
I20240502 14:15:24.693897 27408 raft_consensus.cc:348] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:24.694197 27408 raft_consensus.cc:729] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2bf9b3d134bc4ac79a05d942fc2fb68b, State: Initialized, Role: LEARNER
I20240502 14:15:24.694370 27408 consensus_queue.cc:260] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
W20240502 14:15:24.696754 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.700809 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.706447 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
I20240502 14:15:24.712819 27408 ts_tablet_manager.cc:1398] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20240502 14:15:24.713375 25379 tablet_copy_service.cc:342] P c91321c0a57d4bec8eb3bfba60cee06e: Request end of tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-537b91a6ab124022a1cdfb077055d06f received from {username='slave'} at 127.21.58.69:35541
I20240502 14:15:24.713507 25379 tablet_copy_service.cc:433] P c91321c0a57d4bec8eb3bfba60cee06e: ending tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-537b91a6ab124022a1cdfb077055d06f on tablet 537b91a6ab124022a1cdfb077055d06f with peer 2bf9b3d134bc4ac79a05d942fc2fb68b
W20240502 14:15:24.714164 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.722810 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.734443 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.748104 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.761054 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.777348 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.815582 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.831506 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.848462 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.865831 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.881215 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.926060 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.947811 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:24.967677 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
I20240502 14:15:24.982105 26939 raft_consensus.cc:1203] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.74->[1.75-1.99] Dedup: 1.82->[1.83-1.99]
W20240502 14:15:24.989003 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:25.012725 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
W20240502 14:15:25.036660 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
I20240502 14:15:25.057775 27169 raft_consensus.cc:1050] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e: attempting to promote NON_VOTER 103df70e837147d9a992da0e46ece12e to VOTER
I20240502 14:15:25.058671 27169 consensus_queue.cc:237] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } }
I20240502 14:15:25.060424 26939 raft_consensus.cc:1261] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [term 1 LEARNER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 14:15:25.061461 26440 consensus_queue.cc:1035] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 14:15:25.065042 27103 raft_consensus.cc:2932] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 75 to 102, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:25.065654 26939 raft_consensus.cc:2932] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 75 to 102, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:25.070562 25079 catalog_manager.cc:5484] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index 75 to 102, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:15:25.076651 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Service unavailable: leader transfer in progress
I20240502 14:15:25.077242 25489 raft_consensus.cc:1261] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 14:15:25.078420 26440 consensus_queue.cc:1035] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 14:15:25.086207 27040 raft_consensus.cc:979] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: : Instructing follower 2bf9b3d134bc4ac79a05d942fc2fb68b to start an election
I20240502 14:15:25.086333 27040 raft_consensus.cc:1067] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Signalling peer 2bf9b3d134bc4ac79a05d942fc2fb68b to start an election
I20240502 14:15:25.088914 26676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ead87caf4f9c4695a9c1b8b64852cabf"
dest_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b"
from {username='slave'} at 127.21.58.67:60101
I20240502 14:15:25.089200 26676 raft_consensus.cc:482] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:25.089403 26676 raft_consensus.cc:3037] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.092564 26676 raft_consensus.cc:504] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:25.096325 25357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead87caf4f9c4695a9c1b8b64852cabf" candidate_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:25.096547 25357 raft_consensus.cc:3037] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.089298 25488 raft_consensus.cc:2932] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 75 to 102, 103df70e837147d9a992da0e46ece12e (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:25.103030 25357 raft_consensus.cc:2454] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9b3d134bc4ac79a05d942fc2fb68b in term 2.
I20240502 14:15:25.103669 26610 leader_election.cc:302] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9b3d134bc4ac79a05d942fc2fb68b, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:25.104503 27445 raft_consensus.cc:2781] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:25.104547 26676 leader_election.cc:288] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [CANDIDATE]: Term 2 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323), 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:25.107853 25489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead87caf4f9c4695a9c1b8b64852cabf" candidate_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
I20240502 14:15:25.108065 25489 raft_consensus.cc:3032] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:25.108145 25489 raft_consensus.cc:729] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:25.108425 25489 consensus_queue.cc:260] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:25.108824 25489 raft_consensus.cc:3037] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.112754 25489 raft_consensus.cc:2454] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9b3d134bc4ac79a05d942fc2fb68b in term 2.
W20240502 14:15:25.123409 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Illegal state: replica 1360ad293a3e4ecebf03dbf14cca9cb4 is not leader of this config: current role FOLLOWER
I20240502 14:15:25.181703 26675 raft_consensus.cc:1203] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.73->[1.74-1.109] Dedup: 1.78->[1.79-1.109]
I20240502 14:15:25.210810 27445 raft_consensus.cc:686] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 LEADER]: Becoming Leader. State: Replica: 2bf9b3d134bc4ac79a05d942fc2fb68b, State: Running, Role: LEADER
I20240502 14:15:25.212718 27445 consensus_queue.cc:237] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
W20240502 14:15:25.239712 25317 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58392: Illegal state: replica c91321c0a57d4bec8eb3bfba60cee06e is not leader of this config: current role FOLLOWER
I20240502 14:15:25.263767 25357 consensus_queue.cc:237] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } }
I20240502 14:15:25.267655 26941 raft_consensus.cc:1261] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 14:15:25.268496 27169 consensus_queue.cc:1035] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 14:15:25.273090 25078 catalog_manager.cc:5484] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: term changed from 1 to 2, leader changed from 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) to 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69). New cstate: current_term: 2 leader_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:15:25.279897 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53986: Illegal state: replica 1360ad293a3e4ecebf03dbf14cca9cb4 is not leader of this config: current role FOLLOWER
I20240502 14:15:25.279951 26440 raft_consensus.cc:2932] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 102 to 106, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:25.280733 26939 raft_consensus.cc:2932] T 291df3cd98de4152ab6aaf069bc45a36 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 102 to 106, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } }
I20240502 14:15:25.285514 25071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 291df3cd98de4152ab6aaf069bc45a36 with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:25.287375 25081 catalog_manager.cc:5484] T 291df3cd98de4152ab6aaf069bc45a36 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index 102 to 106, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:15:25.311342 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58392: Illegal state: replica c91321c0a57d4bec8eb3bfba60cee06e is not leader of this config: current role FOLLOWER
I20240502 14:15:25.334718 25469 tablet_service.cc:1514] Processing DeleteTablet for tablet 291df3cd98de4152ab6aaf069bc45a36 with delete_type TABLET_DATA_TOMBSTONED (TS 1360ad293a3e4ecebf03dbf14cca9cb4 not found in new config with opid_index 106) from {username='slave'} at 127.0.0.1:40350
I20240502 14:15:25.335608 27485 tablet_replica.cc:330] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: stopping tablet replica
I20240502 14:15:25.335872 27485 raft_consensus.cc:2229] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:25.349727 27485 raft_consensus.cc:2258] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:25.356447 27485 ts_tablet_manager.cc:1875] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:25.379235 27485 ts_tablet_manager.cc:1888] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.105
I20240502 14:15:25.379441 27485 log.cc:1196] T 291df3cd98de4152ab6aaf069bc45a36 P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/291df3cd98de4152ab6aaf069bc45a36
I20240502 14:15:25.380640 25072 catalog_manager.cc:4830] TS 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255): tablet 291df3cd98de4152ab6aaf069bc45a36 (table rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]) successfully deleted
I20240502 14:15:25.411263 25488 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681"
dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46308
I20240502 14:15:25.411453 25488 raft_consensus.cc:595] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:15:25.421609 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.425770 25488 raft_consensus.cc:1261] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Refusing update from remote peer 2bf9b3d134bc4ac79a05d942fc2fb68b: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240502 14:15:25.429209 25357 raft_consensus.cc:1261] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Refusing update from remote peer 2bf9b3d134bc4ac79a05d942fc2fb68b: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240502 14:15:25.429941 27445 consensus_queue.cc:1035] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 14:15:25.433215 27445 consensus_queue.cc:1035] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
W20240502 14:15:25.437696 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
W20240502 14:15:25.453346 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
W20240502 14:15:25.476503 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.492322 26440 raft_consensus.cc:1050] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: attempting to promote NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b to VOTER
I20240502 14:15:25.493026 26440 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:25.497113 25488 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
W20240502 14:15:25.497264 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.497840 27103 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 14:15:25.502839 26674 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 LEARNER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 14:15:25.503875 27168 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 14:15:25.505340 27168 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index 74 to 114, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:25.507130 25488 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 74 to 114, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:25.510983 26674 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 74 to 114, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
W20240502 14:15:25.511683 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.518738 25081 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 74 to 114, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
W20240502 14:15:25.530107 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.545110 25356 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:25.546573 26675 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240502 14:15:25.549619 27169 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20240502 14:15:25.551064 26677 consensus_queue.cc:237] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 2.114, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:25.553062 27103 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index 114 to 116, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:25.554116 25356 raft_consensus.cc:1261] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Refusing update from remote peer 2bf9b3d134bc4ac79a05d942fc2fb68b: Log matching property violated. Preceding OpId in replica: term: 2 index: 114. Preceding OpId from leader: term: 2 index: 115. (index mismatch)
I20240502 14:15:25.554637 27445 consensus_queue.cc:1035] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20240502 14:15:25.557199 27445 raft_consensus.cc:2932] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 LEADER]: Committing config change with OpId 2.115: config changed from index 104 to 115, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:25.560874 25356 raft_consensus.cc:2932] T ead87caf4f9c4695a9c1b8b64852cabf P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Committing config change with OpId 2.115: config changed from index 104 to 115, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
W20240502 14:15:25.563484 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.564965 25074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ead87caf4f9c4695a9c1b8b64852cabf with cas_config_opid_index 104: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:25.566648 25080 catalog_manager.cc:5484] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 104 to 115, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New cstate: current_term: 2 leader_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:25.574555 25469 tablet_service.cc:1514] Processing DeleteTablet for tablet ead87caf4f9c4695a9c1b8b64852cabf with delete_type TABLET_DATA_TOMBSTONED (TS 1360ad293a3e4ecebf03dbf14cca9cb4 not found in new config with opid_index 115) from {username='slave'} at 127.0.0.1:40350
I20240502 14:15:25.575942 27485 tablet_replica.cc:330] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: stopping tablet replica
I20240502 14:15:25.576128 27485 raft_consensus.cc:2229] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:25.576282 27485 pending_rounds.cc:62] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Trying to abort 1 pending ops.
I20240502 14:15:25.576333 27485 pending_rounds.cc:68] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Aborting op as it isn't in flight: id { term: 2 index: 114 } timestamp: 7023244597419802624 op_type: WRITE_OP write_request { tablet_id: "ead87caf4f9c4695a9c1b8b64852cabf" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000w\266\210i\377sfX\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""30c2ddc692dc5800" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023244597315698688 authz_token { token_data: "\010\242\302\316\261\006\"3\n\005slave\022*\n b9d2e585423c419ab821f0467fb014d3\020\001\030\001 \001(\001" signature: "<redacted>""\002\353\374\354\2418p\260\001\222\002\215\300\226\232\217\252\242\360\373\336[Ti7\260\246@)?Cd\245\344_\365\335\351=\316\367\313H\234\264\220ym\347\332~A\362\267X8\304\037\342i>\246K\210" signing_key_seq_num: 0 } } request_id { client_id: "de48ec247cd54c9cbc7776ccc6b62e14" seq_no: 318 first_incomplete_seq_no: 318 attempt_no: 0 }
I20240502 14:15:25.576864 27485 raft_consensus.cc:2258] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:25.578100 27485 ts_tablet_manager.cc:1875] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:25.580135 25071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 537b91a6ab124022a1cdfb077055d06f with cas_config_opid_index 114: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:25.580880 26675 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 114 to 116, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:25.581549 25081 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index 114 to 116, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:25.585186 25469 tablet_service.cc:1514] Processing DeleteTablet for tablet 537b91a6ab124022a1cdfb077055d06f with delete_type TABLET_DATA_TOMBSTONED (TS 1360ad293a3e4ecebf03dbf14cca9cb4 not found in new config with opid_index 116) from {username='slave'} at 127.0.0.1:40350
W20240502 14:15:25.596264 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.610446 27485 ts_tablet_manager.cc:1888] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.114
I20240502 14:15:25.610617 27485 log.cc:1196] T ead87caf4f9c4695a9c1b8b64852cabf P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/ead87caf4f9c4695a9c1b8b64852cabf
I20240502 14:15:25.611382 27485 tablet_replica.cc:330] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: stopping tablet replica
I20240502 14:15:25.611595 27485 raft_consensus.cc:2229] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:25.611788 27485 pending_rounds.cc:62] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Trying to abort 1 pending ops.
I20240502 14:15:25.611871 27485 pending_rounds.cc:68] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Aborting op as it isn't in flight: id { term: 1 index: 115 } timestamp: 7023244597381505024 op_type: WRITE_OP write_request { tablet_id: "537b91a6ab124022a1cdfb077055d06f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000w\217i\031%\277\262\267\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""076af21bfc79de7c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023244597254967296 authz_token { token_data: "\010\242\302\316\261\006\"3\n\005slave\022*\n 9084066a83c244fca75e98ecb64620f4\020\001\030\001 \001(\001" signature: "<redacted>""\302\2039)&\023\324\004\214\206\313\276i\257\356\370\267Zu\022K) i\306\241l\230\314\006]\037vR\354\032V\260x\315I\020\262\'\030\006\307\006\020\261^h-a:\321\242%\237\220`\371T\266" signing_key_seq_num: 0 } } request_id { client_id: "6e7a7b072bff4cb8bf6fe025d5d11a1d" seq_no: 322 first_incomplete_seq_no: 322 attempt_no: 0 }
I20240502 14:15:25.612115 25072 catalog_manager.cc:4830] TS 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255): tablet ead87caf4f9c4695a9c1b8b64852cabf (table rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]) successfully deleted
I20240502 14:15:25.612427 27485 raft_consensus.cc:2258] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:25.619195 27485 ts_tablet_manager.cc:1875] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:25.629195 27485 ts_tablet_manager.cc:1888] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.115
I20240502 14:15:25.632438 27485 log.cc:1196] T 537b91a6ab124022a1cdfb077055d06f P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/537b91a6ab124022a1cdfb077055d06f
W20240502 14:15:25.633618 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.633733 25072 catalog_manager.cc:4830] TS 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255): tablet 537b91a6ab124022a1cdfb077055d06f (table rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]) successfully deleted
W20240502 14:15:25.648000 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
W20240502 14:15:25.674336 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
W20240502 14:15:25.688975 25447 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
W20240502 14:15:25.707690 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.720657 27205 raft_consensus.cc:979] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: : Instructing follower a5cb74a14a8d434e986ac7d988731e8a to start an election
I20240502 14:15:25.720832 27205 raft_consensus.cc:1067] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Signalling peer a5cb74a14a8d434e986ac7d988731e8a to start an election
I20240502 14:15:25.721534 25227 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681"
dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
from {username='slave'} at 127.21.58.67:58235
I20240502 14:15:25.721791 25227 raft_consensus.cc:482] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:25.721937 25227 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.725898 25227 raft_consensus.cc:504] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
W20240502 14:15:25.727197 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Service unavailable: leader transfer in progress
I20240502 14:15:25.729593 25227 leader_election.cc:288] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 2 election: Requested vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255), c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117)
I20240502 14:15:25.730079 26546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 124 } } ignore_live_leader: true dest_uuid: "c69e4233b679437b8eb3b811680fa196"
I20240502 14:15:25.730410 26546 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.733170 26546 raft_consensus.cc:2454] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 2.
I20240502 14:15:25.733599 25488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681" candidate_uuid: "a5cb74a14a8d434e986ac7d988731e8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 124 } } ignore_live_leader: true dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
I20240502 14:15:25.733682 25164 leader_election.cc:302] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c69e4233b679437b8eb3b811680fa196; no voters:
I20240502 14:15:25.733795 25488 raft_consensus.cc:3032] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:25.733856 25488 raft_consensus.cc:729] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:25.733973 27509 raft_consensus.cc:2781] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:25.734103 27509 raft_consensus.cc:686] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Becoming Leader. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:25.734102 25488 consensus_queue.cc:260] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 124, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:25.734488 25488 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.735347 27509 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 124, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:25.737160 25488 raft_consensus.cc:2454] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb74a14a8d434e986ac7d988731e8a in term 2.
I20240502 14:15:25.738713 25081 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: term changed from 1 to 2, leader changed from 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) to a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65). New cstate: current_term: 2 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:25.749974 25449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53970: Illegal state: replica 1360ad293a3e4ecebf03dbf14cca9cb4 is not leader of this config: current role FOLLOWER
I20240502 14:15:25.780761 26546 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 2 index: 126. (index mismatch)
I20240502 14:15:25.780839 25488 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 2 index: 126. (index mismatch)
I20240502 14:15:25.781584 27151 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 14:15:25.781845 27504 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 14:15:25.799309 27356 mvcc.cc:205] Tried to move back new op lower bound from 7023244598382219264 to 7023244598215151616. Current Snapshot: MvccSnapshot[applied={T|T < 7023244598382219264 or (T in {7023244598382219264})}]
I20240502 14:15:25.803125 25228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de14694a11e14f6082bba7fa70f51553"
dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37394
I20240502 14:15:25.803362 25228 raft_consensus.cc:595] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:25.803905 25227 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 2.127, Last appended by leader: 124, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:25.812909 26545 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 127. Preceding OpId from leader: term: 2 index: 128. (index mismatch)
I20240502 14:15:25.813745 27509 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20240502 14:15:25.827203 27509 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Committing config change with OpId 2.128: config changed from index 86 to 128, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:25.835166 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e30b47ad42124444aadeabc9924e8681 with cas_config_opid_index 86: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:25.836759 26545 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Committing config change with OpId 2.128: config changed from index 86 to 128, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:25.854909 25081 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 86 to 128, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New cstate: current_term: 2 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:25.861519 25469 tablet_service.cc:1514] Processing DeleteTablet for tablet e30b47ad42124444aadeabc9924e8681 with delete_type TABLET_DATA_TOMBSTONED (TS 1360ad293a3e4ecebf03dbf14cca9cb4 not found in new config with opid_index 128) from {username='slave'} at 127.0.0.1:40350
I20240502 14:15:25.861872 27485 tablet_replica.cc:330] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: stopping tablet replica
I20240502 14:15:25.862075 27485 raft_consensus.cc:2229] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:25.862311 27485 raft_consensus.cc:2258] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:25.863139 27485 ts_tablet_manager.cc:1875] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:25.889271 27485 ts_tablet_manager.cc:1888] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.126
I20240502 14:15:25.889422 27485 log.cc:1196] T e30b47ad42124444aadeabc9924e8681 P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/e30b47ad42124444aadeabc9924e8681
I20240502 14:15:25.890493 25072 catalog_manager.cc:4830] TS 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255): tablet e30b47ad42124444aadeabc9924e8681 (table rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]) successfully deleted
I20240502 14:15:25.998509 27379 raft_consensus.cc:979] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: : Instructing follower c69e4233b679437b8eb3b811680fa196 to start an election
I20240502 14:15:25.998687 27379 raft_consensus.cc:1067] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Signalling peer c69e4233b679437b8eb3b811680fa196 to start an election
I20240502 14:15:25.999679 26545 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de14694a11e14f6082bba7fa70f51553"
dest_uuid: "c69e4233b679437b8eb3b811680fa196"
from {username='slave'} at 127.21.58.65:44641
I20240502 14:15:26.000033 26545 raft_consensus.cc:482] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:26.000169 26545 raft_consensus.cc:3037] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.013568 26545 raft_consensus.cc:504] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:26.057416 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de14694a11e14f6082bba7fa70f51553" candidate_uuid: "c69e4233b679437b8eb3b811680fa196" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:26.057653 25228 raft_consensus.cc:3032] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:26.057770 25228 raft_consensus.cc:729] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:26.058089 26545 leader_election.cc:288] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [CANDIDATE]: Term 2 election: Requested vote from peers 103df70e837147d9a992da0e46ece12e (127.21.58.71:37231), a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:26.058137 25228 consensus_queue.cc:260] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:26.058528 25228 raft_consensus.cc:3037] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.080227 25228 raft_consensus.cc:2454] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c69e4233b679437b8eb3b811680fa196 in term 2.
I20240502 14:15:26.080857 26478 leader_election.cc:302] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c69e4233b679437b8eb3b811680fa196; no voters:
I20240502 14:15:26.081386 27518 raft_consensus.cc:2781] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.093061 26939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de14694a11e14f6082bba7fa70f51553" candidate_uuid: "c69e4233b679437b8eb3b811680fa196" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "103df70e837147d9a992da0e46ece12e"
I20240502 14:15:26.093410 26939 raft_consensus.cc:3037] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.095942 26939 raft_consensus.cc:2454] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c69e4233b679437b8eb3b811680fa196 in term 2.
I20240502 14:15:26.108673 27518 raft_consensus.cc:686] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 2 LEADER]: Becoming Leader. State: Replica: c69e4233b679437b8eb3b811680fa196, State: Running, Role: LEADER
I20240502 14:15:26.109264 27518 consensus_queue.cc:237] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:26.190313 25081 catalog_manager.cc:5484] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 reported cstate change: term changed from 1 to 2, leader changed from a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) to c69e4233b679437b8eb3b811680fa196 (127.21.58.68). New cstate: current_term: 2 leader_uuid: "c69e4233b679437b8eb3b811680fa196" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.726949 26939 raft_consensus.cc:1261] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 2 FOLLOWER]: Refusing update from remote peer c69e4233b679437b8eb3b811680fa196: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:26.727627 27579 consensus_queue.cc:1035] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:26.748383 25228 raft_consensus.cc:1261] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Refusing update from remote peer c69e4233b679437b8eb3b811680fa196: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:26.749409 27581 consensus_queue.cc:1035] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:26.834551 26546 consensus_queue.cc:237] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:26.869110 26939 raft_consensus.cc:1261] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 2 FOLLOWER]: Refusing update from remote peer c69e4233b679437b8eb3b811680fa196: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:26.873520 27584 consensus_queue.cc:1035] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:26.897636 27581 raft_consensus.cc:2932] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:26.901397 25074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de14694a11e14f6082bba7fa70f51553 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:26.910312 26939 raft_consensus.cc:2932] T de14694a11e14f6082bba7fa70f51553 P 103df70e837147d9a992da0e46ece12e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:26.910725 25081 catalog_manager.cc:5484] T de14694a11e14f6082bba7fa70f51553 P c69e4233b679437b8eb3b811680fa196 reported cstate change: config changed from index 3 to 5, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "c69e4233b679437b8eb3b811680fa196" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.955724 25207 tablet_service.cc:1514] Processing DeleteTablet for tablet de14694a11e14f6082bba7fa70f51553 with delete_type TABLET_DATA_TOMBSTONED (TS a5cb74a14a8d434e986ac7d988731e8a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34208
I20240502 14:15:27.009490 27597 tablet_replica.cc:330] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: stopping tablet replica
I20240502 14:15:27.009831 27597 raft_consensus.cc:2229] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:27.014794 27597 raft_consensus.cc:2258] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:27.020532 27597 ts_tablet_manager.cc:1875] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:27.041779 27597 ts_tablet_manager.cc:1888] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:15:27.042951 27597 log.cc:1196] T de14694a11e14f6082bba7fa70f51553 P a5cb74a14a8d434e986ac7d988731e8a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/de14694a11e14f6082bba7fa70f51553
I20240502 14:15:27.044011 25072 catalog_manager.cc:4830] TS a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): tablet de14694a11e14f6082bba7fa70f51553 (table TestTable [id=0445c4e8449e45c0889651a22874bc5e]) successfully deleted
I20240502 14:15:27.848788 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:27.884447 26657 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:27.894402 26526 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:27.917848 25339 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:27.962041 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:27.989912 26787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:28.020893 25207 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:28.265590 25228 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 2.176, Last appended by leader: 124, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:28.279562 26544 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 176. Preceding OpId from leader: term: 2 index: 177. (index mismatch)
I20240502 14:15:28.280164 27525 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 175, Time since last communication: 0.000s
I20240502 14:15:28.286558 27509 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Committing config change with OpId 2.177: config changed from index 128 to 177, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
I20240502 14:15:28.294939 25081 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 128 to 177, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New cstate: current_term: 2 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:28.299049 26544 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Committing config change with OpId 2.177: config changed from index 128 to 177, NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) added. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } } }
W20240502 14:15:28.348834 25164 consensus_peers.cc:485] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755): Couldn't send request to peer 2bf9b3d134bc4ac79a05d942fc2fb68b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e30b47ad42124444aadeabc9924e8681. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.491784 25357 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 1.175, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:28.495191 26675 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240502 14:15:28.495958 27103 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I20240502 14:15:28.499169 27103 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.176: config changed from index 116 to 176, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
I20240502 14:15:28.508556 26675 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 116 to 176, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
I20240502 14:15:28.515561 25080 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 116 to 176, NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } } }
W20240502 14:15:28.542238 25292 consensus_peers.cc:485] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e -> Peer 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70:35761): Couldn't send request to peer 6365836c0f2b428bbeb4d69b9ceda2fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 537b91a6ab124022a1cdfb077055d06f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.567001 25357 consensus_queue.cc:237] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 1.192, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 193 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:28.574024 25228 raft_consensus.cc:1261] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 192. Preceding OpId from leader: term: 1 index: 193. (index mismatch)
I20240502 14:15:28.577564 27103 consensus_queue.cc:1035] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 193, Last known committed idx: 191, Time since last communication: 0.000s
I20240502 14:15:28.583010 27604 raft_consensus.cc:2932] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.193: config changed from index -1 to 193, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 193 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:28.589388 25228 raft_consensus.cc:2932] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.193: config changed from index -1 to 193, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New config: { opid_index: 193 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } } }
I20240502 14:15:28.595748 25081 catalog_manager.cc:5484] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index -1 to 193, NON_VOTER c69e4233b679437b8eb3b811680fa196 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 193 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:28.653565 25295 consensus_peers.cc:485] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e -> Peer c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117): Couldn't send request to peer c69e4233b679437b8eb3b811680fa196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02f1d29d09964aa3bfb918e985c18340. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.820407 27696 ts_tablet_manager.cc:902] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Initiating tablet copy from peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:28.977010 27696 tablet_copy_client.cc:308] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46045
I20240502 14:15:29.003981 25248 tablet_copy_service.cc:140] P a5cb74a14a8d434e986ac7d988731e8a: Received BeginTabletCopySession request for tablet e30b47ad42124444aadeabc9924e8681 from peer 2bf9b3d134bc4ac79a05d942fc2fb68b ({username='slave'} at 127.21.58.69:32847)
I20240502 14:15:29.004194 25248 tablet_copy_service.cc:161] P a5cb74a14a8d434e986ac7d988731e8a: Beginning new tablet copy session on tablet e30b47ad42124444aadeabc9924e8681 from peer 2bf9b3d134bc4ac79a05d942fc2fb68b at {username='slave'} at 127.21.58.69:32847: session id = 2bf9b3d134bc4ac79a05d942fc2fb68b-e30b47ad42124444aadeabc9924e8681
I20240502 14:15:29.007102 25248 tablet_copy_source_session.cc:205] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:29.009770 27696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30b47ad42124444aadeabc9924e8681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:29.019321 27696 tablet_copy_client.cc:783] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:29.036680 27696 tablet_copy_client.cc:647] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:29.044541 27696 tablet_copy_client.cc:523] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.047343 27704 ts_tablet_manager.cc:902] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: Initiating tablet copy from peer c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:29.051419 27696 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap starting.
I20240502 14:15:29.089896 27704 tablet_copy_client.cc:308] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44323
I20240502 14:15:29.120445 25379 tablet_copy_service.cc:140] P c91321c0a57d4bec8eb3bfba60cee06e: Received BeginTabletCopySession request for tablet 537b91a6ab124022a1cdfb077055d06f from peer 6365836c0f2b428bbeb4d69b9ceda2fe ({username='slave'} at 127.21.58.70:56741)
I20240502 14:15:29.120644 25379 tablet_copy_service.cc:161] P c91321c0a57d4bec8eb3bfba60cee06e: Beginning new tablet copy session on tablet 537b91a6ab124022a1cdfb077055d06f from peer 6365836c0f2b428bbeb4d69b9ceda2fe at {username='slave'} at 127.21.58.70:56741: session id = 6365836c0f2b428bbeb4d69b9ceda2fe-537b91a6ab124022a1cdfb077055d06f
I20240502 14:15:29.134377 25379 tablet_copy_source_session.cc:205] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:29.139693 27704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b91a6ab124022a1cdfb077055d06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:29.152920 27704 tablet_copy_client.cc:783] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:29.153220 27704 tablet_copy_client.cc:647] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:29.170135 27704 tablet_copy_client.cc:523] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.183598 27704 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap starting.
I20240502 14:15:29.208487 27713 ts_tablet_manager.cc:902] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: Initiating tablet copy from peer c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:29.240511 27713 tablet_copy_client.cc:308] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44323
I20240502 14:15:29.323535 25379 tablet_copy_service.cc:140] P c91321c0a57d4bec8eb3bfba60cee06e: Received BeginTabletCopySession request for tablet 02f1d29d09964aa3bfb918e985c18340 from peer c69e4233b679437b8eb3b811680fa196 ({username='slave'} at 127.21.58.68:59493)
I20240502 14:15:29.323729 25379 tablet_copy_service.cc:161] P c91321c0a57d4bec8eb3bfba60cee06e: Beginning new tablet copy session on tablet 02f1d29d09964aa3bfb918e985c18340 from peer c69e4233b679437b8eb3b811680fa196 at {username='slave'} at 127.21.58.68:59493: session id = c69e4233b679437b8eb3b811680fa196-02f1d29d09964aa3bfb918e985c18340
I20240502 14:15:29.326608 25379 tablet_copy_source_session.cc:205] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:29.341269 27713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f1d29d09964aa3bfb918e985c18340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:29.383221 27713 tablet_copy_client.cc:783] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:29.383574 27713 tablet_copy_client.cc:647] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:29.416582 27713 tablet_copy_client.cc:523] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.426443 27713 tablet_bootstrap.cc:492] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: Bootstrap starting.
I20240502 14:15:30.121305 27713 tablet_bootstrap.cc:492] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=886 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:30.121760 27713 tablet_bootstrap.cc:492] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: Bootstrap complete.
I20240502 14:15:30.122089 27713 ts_tablet_manager.cc:1367] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: Time spent bootstrapping tablet: real 0.696s user 0.273s sys 0.015s
I20240502 14:15:30.122967 27713 raft_consensus.cc:348] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 193 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:30.123351 27713 raft_consensus.cc:729] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c69e4233b679437b8eb3b811680fa196, State: Initialized, Role: LEARNER
I20240502 14:15:30.123598 27713 consensus_queue.cc:260] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 193 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: true } }
I20240502 14:15:30.124547 27713 ts_tablet_manager.cc:1398] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:30.130008 25379 tablet_copy_service.cc:342] P c91321c0a57d4bec8eb3bfba60cee06e: Request end of tablet copy session c69e4233b679437b8eb3b811680fa196-02f1d29d09964aa3bfb918e985c18340 received from {username='slave'} at 127.21.58.68:59493
I20240502 14:15:30.130162 25379 tablet_copy_service.cc:433] P c91321c0a57d4bec8eb3bfba60cee06e: ending tablet copy session c69e4233b679437b8eb3b811680fa196-02f1d29d09964aa3bfb918e985c18340 on tablet 02f1d29d09964aa3bfb918e985c18340 with peer c69e4233b679437b8eb3b811680fa196
I20240502 14:15:30.185971 26545 raft_consensus.cc:1203] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.192->[1.193-1.228] Dedup: 1.214->[1.215-1.228]
I20240502 14:15:30.335359 27704 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=192 overwritten=0 applied=191 ignored=0} inserts{seen=873 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:15:30.336092 27704 tablet_bootstrap.cc:492] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: Bootstrap complete.
I20240502 14:15:30.336467 27704 ts_tablet_manager.cc:1367] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent bootstrapping tablet: real 1.153s user 0.267s sys 0.031s
I20240502 14:15:30.337347 27704 raft_consensus.cc:348] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:30.338060 27704 raft_consensus.cc:729] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6365836c0f2b428bbeb4d69b9ceda2fe, State: Initialized, Role: LEARNER
I20240502 14:15:30.338326 27704 consensus_queue.cc:260] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 191, Last appended: 1.192, Last appended by leader: 192, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: true } }
I20240502 14:15:30.370822 27169 raft_consensus.cc:1050] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: attempting to promote NON_VOTER c69e4233b679437b8eb3b811680fa196 to VOTER
I20240502 14:15:30.371062 27704 ts_tablet_manager.cc:1398] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe: Time spent starting tablet: real 0.034s user 0.017s sys 0.014s
I20240502 14:15:30.371611 27169 consensus_queue.cc:237] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:30.374168 25379 tablet_copy_service.cc:342] P c91321c0a57d4bec8eb3bfba60cee06e: Request end of tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-537b91a6ab124022a1cdfb077055d06f received from {username='slave'} at 127.21.58.70:56741
I20240502 14:15:30.374321 25379 tablet_copy_service.cc:433] P c91321c0a57d4bec8eb3bfba60cee06e: ending tablet copy session 6365836c0f2b428bbeb4d69b9ceda2fe-537b91a6ab124022a1cdfb077055d06f on tablet 537b91a6ab124022a1cdfb077055d06f with peer 6365836c0f2b428bbeb4d69b9ceda2fe
I20240502 14:15:30.378723 26545 raft_consensus.cc:1261] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 1 LEARNER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240502 14:15:30.379343 27604 consensus_queue.cc:1035] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 230, Time since last communication: 0.000s
I20240502 14:15:30.382118 25227 raft_consensus.cc:1261] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240502 14:15:30.383096 27168 consensus_queue.cc:1035] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 230, Time since last communication: 0.000s
I20240502 14:15:30.385202 27652 raft_consensus.cc:2932] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index 193 to 232, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:30.385655 25227 raft_consensus.cc:2932] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 193 to 232, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:30.392928 25080 catalog_manager.cc:5484] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 193 to 232, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:30.401005 27696 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap replayed 1/1 log segments. Stats: ops{read=190 overwritten=0 applied=190 ignored=0} inserts{seen=871 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:30.401695 27696 tablet_bootstrap.cc:492] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Bootstrap complete.
I20240502 14:15:30.402140 27696 ts_tablet_manager.cc:1367] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent bootstrapping tablet: real 1.351s user 0.272s sys 0.026s
I20240502 14:15:30.403221 27696 raft_consensus.cc:348] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:30.403646 27696 raft_consensus.cc:729] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2bf9b3d134bc4ac79a05d942fc2fb68b, State: Initialized, Role: LEARNER
I20240502 14:15:30.403960 27696 consensus_queue.cc:260] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 190, Last appended: 2.190, Last appended by leader: 190, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: true } }
I20240502 14:15:30.406546 27696 ts_tablet_manager.cc:1398] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:30.407624 25248 tablet_copy_service.cc:342] P a5cb74a14a8d434e986ac7d988731e8a: Request end of tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-e30b47ad42124444aadeabc9924e8681 received from {username='slave'} at 127.21.58.69:32847
I20240502 14:15:30.407771 25248 tablet_copy_service.cc:433] P a5cb74a14a8d434e986ac7d988731e8a: ending tablet copy session 2bf9b3d134bc4ac79a05d942fc2fb68b-e30b47ad42124444aadeabc9924e8681 on tablet e30b47ad42124444aadeabc9924e8681 with peer 2bf9b3d134bc4ac79a05d942fc2fb68b
I20240502 14:15:30.445734 26805 raft_consensus.cc:1203] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.175->[1.176-1.215] Dedup: 1.192->[1.193-1.215]
I20240502 14:15:30.456672 26546 raft_consensus.cc:2932] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 193 to 232, c69e4233b679437b8eb3b811680fa196 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:30.565855 27168 raft_consensus.cc:1050] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: attempting to promote NON_VOTER 6365836c0f2b428bbeb4d69b9ceda2fe to VOTER
I20240502 14:15:30.571413 27168 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:30.576385 26805 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 LEARNER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240502 14:15:30.578513 26677 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Refusing update from remote peer c91321c0a57d4bec8eb3bfba60cee06e: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240502 14:15:30.598232 27652 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.016s
I20240502 14:15:30.600035 27652 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.001s
I20240502 14:15:30.603873 27652 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 176 to 219, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:30.606675 26677 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 176 to 219, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:30.615307 25078 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index 176 to 219, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:30.633030 26805 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 176 to 219, 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:30.938848 25357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02f1d29d09964aa3bfb918e985c18340"
dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58536
I20240502 14:15:30.939177 25357 raft_consensus.cc:595] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:30.941094 26674 raft_consensus.cc:1203] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.176->[2.177-2.232] Dedup: 2.190->[2.191-2.232]
I20240502 14:15:30.966221 27773 raft_consensus.cc:979] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: : Instructing follower c69e4233b679437b8eb3b811680fa196 to start an election
I20240502 14:15:30.966543 27169 raft_consensus.cc:1067] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Signalling peer c69e4233b679437b8eb3b811680fa196 to start an election
I20240502 14:15:30.976568 26544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02f1d29d09964aa3bfb918e985c18340"
dest_uuid: "c69e4233b679437b8eb3b811680fa196"
from {username='slave'} at 127.21.58.66:41135
I20240502 14:15:30.977039 26544 raft_consensus.cc:482] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:30.977167 26544 raft_consensus.cc:3037] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:30.979207 26544 raft_consensus.cc:504] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:30.979918 26544 leader_election.cc:288] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [CANDIDATE]: Term 2 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045), c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323)
I20240502 14:15:30.981071 25227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f1d29d09964aa3bfb918e985c18340" candidate_uuid: "c69e4233b679437b8eb3b811680fa196" candidate_term: 2 candidate_status { last_received { term: 1 index: 243 } } ignore_live_leader: true dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:30.981269 25227 raft_consensus.cc:3037] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:30.983812 25227 raft_consensus.cc:2454] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c69e4233b679437b8eb3b811680fa196 in term 2.
I20240502 14:15:30.984383 26478 leader_election.cc:302] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cb74a14a8d434e986ac7d988731e8a, c69e4233b679437b8eb3b811680fa196; no voters:
I20240502 14:15:30.984761 27739 raft_consensus.cc:2781] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:31.016803 25356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f1d29d09964aa3bfb918e985c18340" candidate_uuid: "c69e4233b679437b8eb3b811680fa196" candidate_term: 2 candidate_status { last_received { term: 1 index: 243 } } ignore_live_leader: true dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:31.017063 25356 raft_consensus.cc:3032] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:31.017151 25356 raft_consensus.cc:729] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Running, Role: LEADER
I20240502 14:15:31.017786 25356 consensus_queue.cc:260] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 1.243, Last appended by leader: 243, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:31.018328 25356 raft_consensus.cc:3037] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:31.024399 25356 raft_consensus.cc:2454] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c69e4233b679437b8eb3b811680fa196 in term 2.
W20240502 14:15:31.049443 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50126: Illegal state: replica c91321c0a57d4bec8eb3bfba60cee06e is not leader of this config: current role FOLLOWER
I20240502 14:15:31.074502 27739 raft_consensus.cc:686] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 2 LEADER]: Becoming Leader. State: Replica: c69e4233b679437b8eb3b811680fa196, State: Running, Role: LEADER
I20240502 14:15:31.078136 27739 consensus_queue.cc:237] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 1.243, Last appended by leader: 243, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:31.097945 25078 catalog_manager.cc:5484] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 reported cstate change: term changed from 1 to 2, leader changed from c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) to c69e4233b679437b8eb3b811680fa196 (127.21.58.68). New cstate: current_term: 2 leader_uuid: "c69e4233b679437b8eb3b811680fa196" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:15:31.183495 25187 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37468: Illegal state: replica a5cb74a14a8d434e986ac7d988731e8a is not leader of this config: current role FOLLOWER
I20240502 14:15:31.230427 27618 raft_consensus.cc:1050] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: attempting to promote NON_VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b to VOTER
I20240502 14:15:31.236574 27618 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 2.236, Last appended by leader: 124, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:31.240398 26544 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 236. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20240502 14:15:31.242624 27749 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240502 14:15:31.245462 25356 raft_consensus.cc:1261] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Refusing update from remote peer c69e4233b679437b8eb3b811680fa196: Log matching property violated. Preceding OpId in replica: term: 1 index: 243. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
I20240502 14:15:31.246753 25228 raft_consensus.cc:1261] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Refusing update from remote peer c69e4233b679437b8eb3b811680fa196: Log matching property violated. Preceding OpId in replica: term: 1 index: 243. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
I20240502 14:15:31.247362 27739 consensus_queue.cc:1035] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20240502 14:15:31.247911 27739 consensus_queue.cc:1035] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20240502 14:15:31.250910 27749 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Committing config change with OpId 2.237: config changed from index 177 to 237, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:31.258853 26544 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Committing config change with OpId 2.237: config changed from index 177 to 237, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:31.264820 26544 consensus_queue.cc:237] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 2.245, Last appended by leader: 243, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } }
I20240502 14:15:31.264894 25078 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 177 to 237, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:31.269399 25228 raft_consensus.cc:1261] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Refusing update from remote peer c69e4233b679437b8eb3b811680fa196: Log matching property violated. Preceding OpId in replica: term: 2 index: 245. Preceding OpId from leader: term: 2 index: 246. (index mismatch)
I20240502 14:15:31.271164 27794 consensus_queue.cc:1035] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240502 14:15:31.273061 26677 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 LEARNER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 236. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20240502 14:15:31.273592 27794 raft_consensus.cc:2932] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 [term 2 LEADER]: Committing config change with OpId 2.246: config changed from index 232 to 246, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:31.274055 25228 raft_consensus.cc:2932] T 02f1d29d09964aa3bfb918e985c18340 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Committing config change with OpId 2.246: config changed from index 232 to 246, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } }
I20240502 14:15:31.277451 25074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02f1d29d09964aa3bfb918e985c18340 with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:31.280300 25078 catalog_manager.cc:5484] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 reported cstate change: config changed from index 232 to 246, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "c69e4233b679437b8eb3b811680fa196" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:31.289455 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet 02f1d29d09964aa3bfb918e985c18340 with delete_type TABLET_DATA_TOMBSTONED (TS c91321c0a57d4bec8eb3bfba60cee06e not found in new config with opid_index 246) from {username='slave'} at 127.0.0.1:43744
I20240502 14:15:31.290153 27806 tablet_replica.cc:330] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: stopping tablet replica
I20240502 14:15:31.290428 27806 raft_consensus.cc:2229] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:31.290792 27806 raft_consensus.cc:2258] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:31.291895 27806 ts_tablet_manager.cc:1875] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:31.322206 27749 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240502 14:15:31.327731 26675 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Committing config change with OpId 2.237: config changed from index 177 to 237, 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:31.384361 27806 ts_tablet_manager.cc:1888] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.245
I20240502 14:15:31.384583 27806 log.cc:1196] T 02f1d29d09964aa3bfb918e985c18340 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/02f1d29d09964aa3bfb918e985c18340
I20240502 14:15:31.385524 25071 catalog_manager.cc:4830] TS c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): tablet 02f1d29d09964aa3bfb918e985c18340 (table rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]) successfully deleted
I20240502 14:15:31.475919 25356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "537b91a6ab124022a1cdfb077055d06f"
dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58558
I20240502 14:15:31.476155 25356 raft_consensus.cc:595] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Received request to transfer leadership
W20240502 14:15:31.504781 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.512209 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.529084 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.537581 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.564503 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.584612 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.598284 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.612542 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.626994 25227 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681"
dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37492
I20240502 14:15:31.627310 25227 raft_consensus.cc:595] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Received request to transfer leadership
W20240502 14:15:31.629524 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.643360 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.644387 27618 raft_consensus.cc:979] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: : Instructing follower 2bf9b3d134bc4ac79a05d942fc2fb68b to start an election
I20240502 14:15:31.644676 27618 raft_consensus.cc:1067] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Signalling peer 2bf9b3d134bc4ac79a05d942fc2fb68b to start an election
W20240502 14:15:31.657613 25185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54784: Service unavailable: leader transfer in progress
W20240502 14:15:31.659902 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.663050 25185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54784: Service unavailable: leader transfer in progress
I20240502 14:15:31.664988 26677 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681"
dest_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b"
from {username='slave'} at 127.21.58.65:47535
I20240502 14:15:31.665184 26677 raft_consensus.cc:482] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:31.665333 26677 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:15:31.667758 25185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54784: Service unavailable: leader transfer in progress
W20240502 14:15:31.675045 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.678834 25185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54784: Service unavailable: leader transfer in progress
I20240502 14:15:31.678953 26677 raft_consensus.cc:504] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:31.681841 25227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681" candidate_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" candidate_term: 3 candidate_status { last_received { term: 2 index: 248 } } ignore_live_leader: true dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:31.682035 25227 raft_consensus.cc:3032] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:31.682125 25227 raft_consensus.cc:729] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:31.682364 25227 consensus_queue.cc:260] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 2.248, Last appended by leader: 248, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:31.682678 25227 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:31.684219 26677 leader_election.cc:288] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [CANDIDATE]: Term 3 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045), c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117)
W20240502 14:15:31.686259 25185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54784: Illegal state: replica a5cb74a14a8d434e986ac7d988731e8a is not leader of this config: current role FOLLOWER
I20240502 14:15:31.686303 25227 raft_consensus.cc:2454] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9b3d134bc4ac79a05d942fc2fb68b in term 3.
W20240502 14:15:31.690127 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.694005 26611 leader_election.cc:302] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9b3d134bc4ac79a05d942fc2fb68b, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:31.694308 27445 raft_consensus.cc:2781] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:31.694451 27445 raft_consensus.cc:686] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 3 LEADER]: Becoming Leader. State: Replica: 2bf9b3d134bc4ac79a05d942fc2fb68b, State: Running, Role: LEADER
I20240502 14:15:31.694722 27445 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 2.248, Last appended by leader: 248, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:31.700368 25078 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: term changed from 2 to 3, leader changed from a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) to 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69). New cstate: current_term: 3 leader_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:15:31.709453 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.711952 26546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30b47ad42124444aadeabc9924e8681" candidate_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" candidate_term: 3 candidate_status { last_received { term: 2 index: 248 } } ignore_live_leader: true dest_uuid: "c69e4233b679437b8eb3b811680fa196"
I20240502 14:15:31.712158 26546 raft_consensus.cc:3037] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:31.714659 26546 raft_consensus.cc:2454] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9b3d134bc4ac79a05d942fc2fb68b in term 3.
W20240502 14:15:31.734485 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.751346 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.763046 25227 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 3 FOLLOWER]: Refusing update from remote peer 2bf9b3d134bc4ac79a05d942fc2fb68b: Log matching property violated. Preceding OpId in replica: term: 2 index: 248. Preceding OpId from leader: term: 3 index: 250. (index mismatch)
I20240502 14:15:31.763268 26544 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 3 FOLLOWER]: Refusing update from remote peer 2bf9b3d134bc4ac79a05d942fc2fb68b: Log matching property violated. Preceding OpId in replica: term: 2 index: 248. Preceding OpId from leader: term: 3 index: 250. (index mismatch)
I20240502 14:15:31.764480 27695 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240502 14:15:31.764930 27695 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240502 14:15:31.768503 27340 mvcc.cc:205] Tried to move back new op lower bound from 7023244622896574464 to 7023244622630526976. Current Snapshot: MvccSnapshot[applied={T|T < 7023244622896574464}]
W20240502 14:15:31.770380 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.776489 26677 consensus_queue.cc:237] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 3.250, Last appended by leader: 248, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } }
I20240502 14:15:31.780328 26546 raft_consensus.cc:1261] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 3 FOLLOWER]: Refusing update from remote peer 2bf9b3d134bc4ac79a05d942fc2fb68b: Log matching property violated. Preceding OpId in replica: term: 3 index: 250. Preceding OpId from leader: term: 3 index: 251. (index mismatch)
I20240502 14:15:31.781486 27445 consensus_queue.cc:1035] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240502 14:15:31.784587 27695 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 3 LEADER]: Committing config change with OpId 3.251: config changed from index 237 to 251, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:31.789048 25074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e30b47ad42124444aadeabc9924e8681 with cas_config_opid_index 237: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 14:15:31.792755 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.793048 26546 raft_consensus.cc:2932] T e30b47ad42124444aadeabc9924e8681 P c69e4233b679437b8eb3b811680fa196 [term 3 FOLLOWER]: Committing config change with OpId 3.251: config changed from index 237 to 251, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } }
I20240502 14:15:31.797111 25081 catalog_manager.cc:5484] T e30b47ad42124444aadeabc9924e8681 P 2bf9b3d134bc4ac79a05d942fc2fb68b reported cstate change: config changed from index 237 to 251, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New cstate: current_term: 3 leader_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:31.804039 25207 tablet_service.cc:1514] Processing DeleteTablet for tablet e30b47ad42124444aadeabc9924e8681 with delete_type TABLET_DATA_TOMBSTONED (TS a5cb74a14a8d434e986ac7d988731e8a not found in new config with opid_index 251) from {username='slave'} at 127.0.0.1:34208
I20240502 14:15:31.804713 27827 tablet_replica.cc:330] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: stopping tablet replica
I20240502 14:15:31.804924 27827 raft_consensus.cc:2229] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:31.805191 27827 raft_consensus.cc:2258] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:31.806066 27827 ts_tablet_manager.cc:1875] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:31.815376 27827 ts_tablet_manager.cc:1888] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.250
I20240502 14:15:31.815549 27827 log.cc:1196] T e30b47ad42124444aadeabc9924e8681 P a5cb74a14a8d434e986ac7d988731e8a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/e30b47ad42124444aadeabc9924e8681
I20240502 14:15:31.816568 25072 catalog_manager.cc:4830] TS a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): tablet e30b47ad42124444aadeabc9924e8681 (table rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]) successfully deleted
W20240502 14:15:31.817767 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.842296 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.869803 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.893483 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.918571 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
W20240502 14:15:31.948953 25317 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Service unavailable: leader transfer in progress
I20240502 14:15:31.952293 27169 raft_consensus.cc:979] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: : Instructing follower 6365836c0f2b428bbeb4d69b9ceda2fe to start an election
I20240502 14:15:31.952454 27169 raft_consensus.cc:1067] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Signalling peer 6365836c0f2b428bbeb4d69b9ceda2fe to start an election
I20240502 14:15:31.954846 26805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "537b91a6ab124022a1cdfb077055d06f"
dest_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe"
from {username='slave'} at 127.21.58.66:49679
I20240502 14:15:31.955168 26805 raft_consensus.cc:482] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:31.955389 26805 raft_consensus.cc:3037] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:31.958288 26805 raft_consensus.cc:504] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:31.962381 25357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b91a6ab124022a1cdfb077055d06f" candidate_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 242 } } ignore_live_leader: true dest_uuid: "c91321c0a57d4bec8eb3bfba60cee06e"
I20240502 14:15:31.962610 25357 raft_consensus.cc:3032] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:31.962728 25357 raft_consensus.cc:729] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c91321c0a57d4bec8eb3bfba60cee06e, State: Running, Role: LEADER
I20240502 14:15:31.963119 25357 consensus_queue.cc:260] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.242, Last appended by leader: 242, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:31.963893 25357 raft_consensus.cc:3037] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:31.964751 26805 leader_election.cc:288] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [CANDIDATE]: Term 2 election: Requested vote from peers c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323), 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755)
I20240502 14:15:31.968271 25357 raft_consensus.cc:2454] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6365836c0f2b428bbeb4d69b9ceda2fe in term 2.
W20240502 14:15:31.982272 25319 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50148: Illegal state: replica c91321c0a57d4bec8eb3bfba60cee06e is not leader of this config: current role FOLLOWER
I20240502 14:15:31.985953 26740 leader_election.cc:302] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6365836c0f2b428bbeb4d69b9ceda2fe, c91321c0a57d4bec8eb3bfba60cee06e; no voters:
I20240502 14:15:31.993652 27841 raft_consensus.cc:2781] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:32.010202 27841 raft_consensus.cc:686] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 2 LEADER]: Becoming Leader. State: Replica: 6365836c0f2b428bbeb4d69b9ceda2fe, State: Running, Role: LEADER
I20240502 14:15:32.010991 27841 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.242, Last appended by leader: 242, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:32.020193 25081 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe reported cstate change: term changed from 1 to 2, leader changed from c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) to 6365836c0f2b428bbeb4d69b9ceda2fe (127.21.58.70). New cstate: current_term: 2 leader_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:32.026288 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b91a6ab124022a1cdfb077055d06f" candidate_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 242 } } ignore_live_leader: true dest_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b"
I20240502 14:15:32.026589 26677 raft_consensus.cc:3037] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:32.030408 26677 raft_consensus.cc:2454] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6365836c0f2b428bbeb4d69b9ceda2fe in term 2.
I20240502 14:15:32.165050 25359 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Refusing update from remote peer 6365836c0f2b428bbeb4d69b9ceda2fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 2 index: 244. (index mismatch)
I20240502 14:15:32.166045 27842 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20240502 14:15:32.173606 27777 mvcc.cc:205] Tried to move back new op lower bound from 7023244624534380544 to 7023244623923085312. Current Snapshot: MvccSnapshot[applied={T|T < 7023244624534380544}]
I20240502 14:15:32.182765 26677 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Refusing update from remote peer 6365836c0f2b428bbeb4d69b9ceda2fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 2 index: 244. (index mismatch)
I20240502 14:15:32.192014 27841 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20240502 14:15:32.323361 26804 consensus_queue.cc:237] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 2.246, Last appended by leader: 242, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } }
I20240502 14:15:32.327241 26677 raft_consensus.cc:1261] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Refusing update from remote peer 6365836c0f2b428bbeb4d69b9ceda2fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 246. Preceding OpId from leader: term: 2 index: 247. (index mismatch)
I20240502 14:15:32.327840 27842 consensus_queue.cc:1035] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20240502 14:15:32.333711 27842 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe [term 2 LEADER]: Committing config change with OpId 2.247: config changed from index 219 to 247, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:32.337950 26674 raft_consensus.cc:2932] T 537b91a6ab124022a1cdfb077055d06f P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 2 FOLLOWER]: Committing config change with OpId 2.247: config changed from index 219 to 247, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } } }
I20240502 14:15:32.340183 25078 catalog_manager.cc:5484] T 537b91a6ab124022a1cdfb077055d06f P 6365836c0f2b428bbeb4d69b9ceda2fe reported cstate change: config changed from index 219 to 247, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6365836c0f2b428bbeb4d69b9ceda2fe" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 35761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:32.348193 25071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 537b91a6ab124022a1cdfb077055d06f with cas_config_opid_index 219: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:32.349763 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet 537b91a6ab124022a1cdfb077055d06f with delete_type TABLET_DATA_TOMBSTONED (TS c91321c0a57d4bec8eb3bfba60cee06e not found in new config with opid_index 247) from {username='slave'} at 127.0.0.1:43744
I20240502 14:15:32.351781 27864 tablet_replica.cc:330] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: stopping tablet replica
I20240502 14:15:32.352030 27864 raft_consensus.cc:2229] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:32.352808 27864 raft_consensus.cc:2258] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:32.353780 27864 ts_tablet_manager.cc:1875] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:32.369766 27864 ts_tablet_manager.cc:1888] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.246
I20240502 14:15:32.369988 27864 log.cc:1196] T 537b91a6ab124022a1cdfb077055d06f P c91321c0a57d4bec8eb3bfba60cee06e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/537b91a6ab124022a1cdfb077055d06f
I20240502 14:15:32.370726 25071 catalog_manager.cc:4830] TS c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): tablet 537b91a6ab124022a1cdfb077055d06f (table rebalance_test_table_4 [id=9084066a83c244fca75e98ecb64620f4]) successfully deleted
I20240502 14:15:33.033236 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:33.302937 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:33.384588 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:33.404650 25207 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:33.475104 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:33.546622 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:33.578652 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:15:33.985306 26613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.68:43117, user_credentials={real_user=slave}} blocked reactor thread for 187692us
I20240502 14:15:34.598785 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:34.616117 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:34.704438 25207 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:34.725342 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:34.738765 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:34.756069 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:34.781132 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:36.715696 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:36.762697 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:36.840724 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:36.864161 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:36.952832 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:36.967638 25207 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:37.096946 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:38.844820 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:38.883160 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:38.898648 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:38.959726 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:38.961834 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:38.963239 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:39.079789 25207 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:40.036396 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:40.063323 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:40.073236 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:40.079406 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:40.096107 25207 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:40.123474 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:40.147747 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9ab9bceb14584c9b9819e55247c54100 | 127.21.58.126:40155 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:45959 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
103df70e837147d9a992da0e46ece12e | 127.21.58.71:37231 | HEALTHY | <none> | 0 | 0
1360ad293a3e4ecebf03dbf14cca9cb4 | 127.21.58.67:33255 | HEALTHY | <none> | 5 | 0
2bf9b3d134bc4ac79a05d942fc2fb68b | 127.21.58.69:44755 | HEALTHY | <none> | 2 | 0
6365836c0f2b428bbeb4d69b9ceda2fe | 127.21.58.70:35761 | HEALTHY | <none> | 1 | 0
a5cb74a14a8d434e986ac7d988731e8a | 127.21.58.65:46045 | HEALTHY | <none> | 3 | 0
c69e4233b679437b8eb3b811680fa196 | 127.21.58.68:43117 | HEALTHY | <none> | 2 | 0
c91321c0a57d4bec8eb3bfba60cee06e | 127.21.58.66:44323 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.21.58.65 | experimental | 127.21.58.65:46045
local_ip_for_outbound_sockets | 127.21.58.66 | experimental | 127.21.58.66:44323
local_ip_for_outbound_sockets | 127.21.58.67 | experimental | 127.21.58.67:33255
local_ip_for_outbound_sockets | 127.21.58.68 | experimental | 127.21.58.68:43117
local_ip_for_outbound_sockets | 127.21.58.69 | experimental | 127.21.58.69:44755
local_ip_for_outbound_sockets | 127.21.58.70 | experimental | 127.21.58.70:35761
local_ip_for_outbound_sockets | 127.21.58.71 | experimental | 127.21.58.71:37231
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.58.65:46045
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.58.66:44323
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.58.67:33255
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.21.58.68:43117
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.21.58.69:44755
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.21.58.70:35761
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.21.58.71:37231
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:45959 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:15:40.195847 21737 log_verifier.cc:126] Checking tablet 02f1d29d09964aa3bfb918e985c18340
I20240502 14:15:40.348562 21737 log_verifier.cc:177] Verified matching terms for 436 ops in tablet 02f1d29d09964aa3bfb918e985c18340
I20240502 14:15:40.349117 21737 log_verifier.cc:126] Checking tablet 0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:40.491895 21737 log_verifier.cc:177] Verified matching terms for 390 ops in tablet 0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:40.492535 21737 log_verifier.cc:126] Checking tablet 232f776da8074d5bb2643f4add1cb06a
I20240502 14:15:40.666621 21737 log_verifier.cc:177] Verified matching terms for 403 ops in tablet 232f776da8074d5bb2643f4add1cb06a
I20240502 14:15:40.667141 21737 log_verifier.cc:126] Checking tablet 291df3cd98de4152ab6aaf069bc45a36
I20240502 14:15:40.796825 21737 log_verifier.cc:177] Verified matching terms for 398 ops in tablet 291df3cd98de4152ab6aaf069bc45a36
I20240502 14:15:40.797350 21737 log_verifier.cc:126] Checking tablet 37ae6a3602fb4437a0e7c337494ea8e8
I20240502 14:15:40.970925 21737 log_verifier.cc:177] Verified matching terms for 426 ops in tablet 37ae6a3602fb4437a0e7c337494ea8e8
I20240502 14:15:40.971648 21737 log_verifier.cc:126] Checking tablet 503e0727a7a14863847f375032cb8144
I20240502 14:15:41.129556 21737 log_verifier.cc:177] Verified matching terms for 416 ops in tablet 503e0727a7a14863847f375032cb8144
I20240502 14:15:41.130102 21737 log_verifier.cc:126] Checking tablet 537b91a6ab124022a1cdfb077055d06f
I20240502 14:15:41.287249 21737 log_verifier.cc:177] Verified matching terms for 412 ops in tablet 537b91a6ab124022a1cdfb077055d06f
I20240502 14:15:41.287823 21737 log_verifier.cc:126] Checking tablet 57558d7d2f9943dc8fe5fd127f3062e8
I20240502 14:15:41.435139 21737 log_verifier.cc:177] Verified matching terms for 401 ops in tablet 57558d7d2f9943dc8fe5fd127f3062e8
I20240502 14:15:41.435701 21737 log_verifier.cc:126] Checking tablet 60e291dc59934bc38239bb84b76975c9
I20240502 14:15:41.616734 21737 log_verifier.cc:177] Verified matching terms for 462 ops in tablet 60e291dc59934bc38239bb84b76975c9
I20240502 14:15:41.617312 21737 log_verifier.cc:126] Checking tablet 66c85e449bea4f829fe6d1c79206591d
I20240502 14:15:41.763291 21737 log_verifier.cc:177] Verified matching terms for 384 ops in tablet 66c85e449bea4f829fe6d1c79206591d
I20240502 14:15:41.763808 21737 log_verifier.cc:126] Checking tablet a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:41.931937 21737 log_verifier.cc:177] Verified matching terms for 403 ops in tablet a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:41.932454 21737 log_verifier.cc:126] Checking tablet de14694a11e14f6082bba7fa70f51553
I20240502 14:15:41.949407 21737 log_verifier.cc:177] Verified matching terms for 5 ops in tablet de14694a11e14f6082bba7fa70f51553
I20240502 14:15:41.949513 21737 log_verifier.cc:126] Checking tablet def417ebb8354201bb6c17eed13ebbf7
I20240502 14:15:42.088233 21737 log_verifier.cc:177] Verified matching terms for 347 ops in tablet def417ebb8354201bb6c17eed13ebbf7
I20240502 14:15:42.088703 21737 log_verifier.cc:126] Checking tablet e30b47ad42124444aadeabc9924e8681
I20240502 14:15:42.232096 21737 log_verifier.cc:177] Verified matching terms for 393 ops in tablet e30b47ad42124444aadeabc9924e8681
I20240502 14:15:42.232635 21737 log_verifier.cc:126] Checking tablet ead87caf4f9c4695a9c1b8b64852cabf
I20240502 14:15:42.383877 21737 log_verifier.cc:177] Verified matching terms for 414 ops in tablet ead87caf4f9c4695a9c1b8b64852cabf
I20240502 14:15:42.384460 21737 log_verifier.cc:126] Checking tablet fcc673d4727a4c049228070d9e2ada05
I20240502 14:15:42.542436 21737 log_verifier.cc:177] Verified matching terms for 415 ops in tablet fcc673d4727a4c049228070d9e2ada05
I20240502 14:15:42.545857 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.72:0
--local_ip_for_outbound_sockets=127.21.58.72
--webserver_interface=127.21.58.72
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--builtin_ntp_servers=127.21.58.84:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:15:42.903299 28156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:42.903672 28156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:42.903833 28156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:42.913537 28156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:42.913650 28156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:42.913767 28156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.72
I20240502 14:15:42.923506 28156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.58.72
--webserver_port=0
--tserver_master_addrs=127.21.58.126:40155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.72
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:42.924845 28156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:42.926441 28156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:42.938684 28161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:42.939447 28162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:42.939997 28164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:42.940765 28156 server_base.cc:1034] running on GCE node
I20240502 14:15:42.941324 28156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:42.942173 28156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:42.943362 28156 hybrid_clock.cc:648] HybridClock initialized: now 1714659342943319 us; error 34 us; skew 500 ppm
I20240502 14:15:42.943676 28156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:42.949801 28156 webserver.cc:460] Webserver started at http://127.21.58.72:43189/ using document root <none> and password file <none>
I20240502 14:15:42.950479 28156 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:42.950587 28156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:42.950896 28156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:42.953485 28156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1fe6ac5a1583483eb93f50988c1b22a8"
format_stamp: "Formatted at 2024-05-02 14:15:42 on dist-test-slave-wtgm"
I20240502 14:15:42.954174 28156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1fe6ac5a1583483eb93f50988c1b22a8"
format_stamp: "Formatted at 2024-05-02 14:15:42 on dist-test-slave-wtgm"
I20240502 14:15:42.958772 28156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 14:15:42.962168 28170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:42.963318 28156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:42.963456 28156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1fe6ac5a1583483eb93f50988c1b22a8"
format_stamp: "Formatted at 2024-05-02 14:15:42 on dist-test-slave-wtgm"
I20240502 14:15:42.963619 28156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:42.998061 28156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:42.999020 28156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:42.999333 28156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:43.000293 28156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:43.001672 28156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:43.001755 28156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:43.001868 28156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:43.001956 28156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:43.037276 28156 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.72:36113
I20240502 14:15:43.037338 28282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.72:36113 every 8 connection(s)
I20240502 14:15:43.038497 28156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:15:43.045446 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 28156
I20240502 14:15:43.045642 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:15:43.051883 28283 heartbeater.cc:347] Connected to a master server at 127.21.58.126:40155
I20240502 14:15:43.052146 28283 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:43.052886 28283 heartbeater.cc:511] Master 127.21.58.126:40155 requested a full tablet report, sending...
I20240502 14:15:43.054208 25080 ts_manager.cc:194] Registered new tserver with Master: 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72:36113)
I20240502 14:15:43.055356 25080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.72:38963
I20240502 14:15:43.540438 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:43.565822 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:43.582263 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:43.594123 28218 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:43.598551 25207 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:43.611882 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:43.630793 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:43.645398 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:43.820060 26921 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:43.832587 28218 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:43.839498 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:43.841001 25469 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:43.857574 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:43.869684 25207 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:43.871747 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:43.886833 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:43.974934 25488 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 1.390, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:43.977365 26939 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20240502 14:15:43.978617 28288 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20240502 14:15:43.996395 28288 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.391: config changed from index 98 to 391, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.001602 26939 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 98 to 391, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.002523 25082 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 98 to 391, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:44.005506 25425 consensus_peers.cc:485] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72:36113): Couldn't send request to peer 1fe6ac5a1583483eb93f50988c1b22a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0723f00c11d744f8b244a3d003ce7b7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:44.024338 25227 consensus_queue.cc:237] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 1.347, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:44.026748 25357 raft_consensus.cc:1261] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20240502 14:15:44.027338 28086 consensus_queue.cc:1035] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20240502 14:15:44.030086 28395 raft_consensus.cc:2932] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.348: config changed from index -1 to 348, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.030790 25357 raft_consensus.cc:2932] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index -1 to 348, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.035966 25081 catalog_manager.cc:5484] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e reported cstate change: config changed from index -1 to 348, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
W20240502 14:15:44.040726 25164 consensus_peers.cc:485] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72:36113): Couldn't send request to peer 1fe6ac5a1583483eb93f50988c1b22a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: def417ebb8354201bb6c17eed13ebbf7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:44.051856 25227 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
W20240502 14:15:44.054703 25164 consensus_peers.cc:485] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a -> Peer 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72:36113): Couldn't send request to peer 1fe6ac5a1583483eb93f50988c1b22a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a68699dbcae34cfc94f2559e0516e5e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:44.055308 26544 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 1 index: 404. (index mismatch)
I20240502 14:15:44.055757 28086 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20240502 14:15:44.057444 28283 heartbeater.cc:503] Master 127.21.58.126:40155 was elected leader, sending a full tablet report...
I20240502 14:15:44.058373 28395 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.404: config changed from index 101 to 404, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.058926 26544 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Committing config change with OpId 1.404: config changed from index 101 to 404, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.063067 25081 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 reported cstate change: config changed from index 101 to 404, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.075325 25488 consensus_queue.cc:237] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 462, Committed index: 462, Last appended: 1.462, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
W20240502 14:15:44.077641 25425 consensus_peers.cc:485] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 -> Peer 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72:36113): Couldn't send request to peer 1fe6ac5a1583483eb93f50988c1b22a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60e291dc59934bc38239bb84b76975c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:44.077844 26674 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 462. Preceding OpId from leader: term: 1 index: 463. (index mismatch)
I20240502 14:15:44.078366 28288 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 463, Last known committed idx: 462, Time since last communication: 0.000s
I20240502 14:15:44.080760 28379 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.463: config changed from index 120 to 463, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.081398 26674 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.463: config changed from index 120 to 463, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New config: { opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } } }
I20240502 14:15:44.086722 25084 catalog_manager.cc:5484] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 120 to 463, NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:44.409528 28452 ts_tablet_manager.cc:902] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:44.411409 28452 tablet_copy_client.cc:308] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:44.417393 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet 0723f00c11d744f8b244a3d003ce7b7a from peer 1fe6ac5a1583483eb93f50988c1b22a8 ({username='slave'} at 127.21.58.72:53975)
I20240502 14:15:44.417542 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet 0723f00c11d744f8b244a3d003ce7b7a from peer 1fe6ac5a1583483eb93f50988c1b22a8 at {username='slave'} at 127.21.58.72:53975: session id = 1fe6ac5a1583483eb93f50988c1b22a8-0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:44.419368 25509 tablet_copy_source_session.cc:205] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:44.423583 28452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723f00c11d744f8b244a3d003ce7b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.432018 28452 tablet_copy_client.cc:783] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:44.432272 28452 tablet_copy_client.cc:647] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:44.434878 28452 tablet_copy_client.cc:523] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:44.437525 28452 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap starting.
I20240502 14:15:44.467062 28458 ts_tablet_manager.cc:902] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: Initiating tablet copy from peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:44.469105 28458 tablet_copy_client.cc:308] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46045
I20240502 14:15:44.474784 25248 tablet_copy_service.cc:140] P a5cb74a14a8d434e986ac7d988731e8a: Received BeginTabletCopySession request for tablet a68699dbcae34cfc94f2559e0516e5e8 from peer 1fe6ac5a1583483eb93f50988c1b22a8 ({username='slave'} at 127.21.58.72:34387)
I20240502 14:15:44.474897 25248 tablet_copy_service.cc:161] P a5cb74a14a8d434e986ac7d988731e8a: Beginning new tablet copy session on tablet a68699dbcae34cfc94f2559e0516e5e8 from peer 1fe6ac5a1583483eb93f50988c1b22a8 at {username='slave'} at 127.21.58.72:34387: session id = 1fe6ac5a1583483eb93f50988c1b22a8-a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:44.476758 25248 tablet_copy_source_session.cc:205] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:44.477809 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68699dbcae34cfc94f2559e0516e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.482798 28458 tablet_copy_client.cc:783] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:44.483016 28458 tablet_copy_client.cc:647] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:44.485885 28458 tablet_copy_client.cc:523] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:44.487890 28452 log.cc:824] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:44.489169 28458 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap starting.
I20240502 14:15:44.537613 28464 ts_tablet_manager.cc:902] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: Initiating tablet copy from peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
I20240502 14:15:44.542321 28467 ts_tablet_manager.cc:902] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: Initiating tablet copy from peer 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255)
I20240502 14:15:44.549608 28467 tablet_copy_client.cc:308] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:33255
I20240502 14:15:44.550419 28464 tablet_copy_client.cc:308] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46045
I20240502 14:15:44.550616 25509 tablet_copy_service.cc:140] P 1360ad293a3e4ecebf03dbf14cca9cb4: Received BeginTabletCopySession request for tablet 60e291dc59934bc38239bb84b76975c9 from peer 1fe6ac5a1583483eb93f50988c1b22a8 ({username='slave'} at 127.21.58.72:53975)
I20240502 14:15:44.550772 25509 tablet_copy_service.cc:161] P 1360ad293a3e4ecebf03dbf14cca9cb4: Beginning new tablet copy session on tablet 60e291dc59934bc38239bb84b76975c9 from peer 1fe6ac5a1583483eb93f50988c1b22a8 at {username='slave'} at 127.21.58.72:53975: session id = 1fe6ac5a1583483eb93f50988c1b22a8-60e291dc59934bc38239bb84b76975c9
I20240502 14:15:44.551246 25248 tablet_copy_service.cc:140] P a5cb74a14a8d434e986ac7d988731e8a: Received BeginTabletCopySession request for tablet def417ebb8354201bb6c17eed13ebbf7 from peer 1fe6ac5a1583483eb93f50988c1b22a8 ({username='slave'} at 127.21.58.72:34387)
I20240502 14:15:44.551425 25248 tablet_copy_service.cc:161] P a5cb74a14a8d434e986ac7d988731e8a: Beginning new tablet copy session on tablet def417ebb8354201bb6c17eed13ebbf7 from peer 1fe6ac5a1583483eb93f50988c1b22a8 at {username='slave'} at 127.21.58.72:34387: session id = 1fe6ac5a1583483eb93f50988c1b22a8-def417ebb8354201bb6c17eed13ebbf7
I20240502 14:15:44.553669 25509 tablet_copy_source_session.cc:205] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:44.554289 25248 tablet_copy_source_session.cc:205] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:44.556103 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e291dc59934bc38239bb84b76975c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.556540 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def417ebb8354201bb6c17eed13ebbf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.576792 28464 tablet_copy_client.cc:783] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:44.577091 28464 tablet_copy_client.cc:647] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:44.580060 28467 tablet_copy_client.cc:783] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:44.580557 28464 tablet_copy_client.cc:523] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:44.584300 28464 tablet_bootstrap.cc:492] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap starting.
I20240502 14:15:44.586921 28467 tablet_copy_client.cc:647] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:44.590242 28467 tablet_copy_client.cc:523] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:44.597020 28467 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap starting.
I20240502 14:15:44.982069 28452 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=391 overwritten=0 applied=391 ignored=0} inserts{seen=1019 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:44.982872 28452 tablet_bootstrap.cc:492] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap complete.
I20240502 14:15:44.983605 28452 ts_tablet_manager.cc:1367] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent bootstrapping tablet: real 0.546s user 0.433s sys 0.047s
I20240502 14:15:44.992563 28452 raft_consensus.cc:348] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:44.993173 28452 raft_consensus.cc:729] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fe6ac5a1583483eb93f50988c1b22a8, State: Initialized, Role: LEARNER
I20240502 14:15:44.993804 28452 consensus_queue.cc:260] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 391, Last appended: 1.391, Last appended by leader: 391, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:45.007579 28452 ts_tablet_manager.cc:1398] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent starting tablet: real 0.024s user 0.016s sys 0.004s
I20240502 14:15:45.010473 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-0723f00c11d744f8b244a3d003ce7b7a received from {username='slave'} at 127.21.58.72:53975
I20240502 14:15:45.010690 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-0723f00c11d744f8b244a3d003ce7b7a on tablet 0723f00c11d744f8b244a3d003ce7b7a with peer 1fe6ac5a1583483eb93f50988c1b22a8
I20240502 14:15:45.047108 28458 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=404 overwritten=0 applied=404 ignored=0} inserts{seen=1073 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:45.047636 28458 tablet_bootstrap.cc:492] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap complete.
I20240502 14:15:45.047982 28458 ts_tablet_manager.cc:1367] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent bootstrapping tablet: real 0.559s user 0.433s sys 0.067s
I20240502 14:15:45.048947 28458 raft_consensus.cc:348] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:45.049238 28458 raft_consensus.cc:729] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fe6ac5a1583483eb93f50988c1b22a8, State: Initialized, Role: LEARNER
I20240502 14:15:45.049474 28458 consensus_queue.cc:260] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 404, Last appended: 1.404, Last appended by leader: 404, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:45.050163 28458 ts_tablet_manager.cc:1398] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:45.051203 28238 raft_consensus.cc:1203] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.390->[1.391-1.391] Dedup: 1.391->[]
I20240502 14:15:45.052847 25248 tablet_copy_service.cc:342] P a5cb74a14a8d434e986ac7d988731e8a: Request end of tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-a68699dbcae34cfc94f2559e0516e5e8 received from {username='slave'} at 127.21.58.72:34387
I20240502 14:15:45.052997 25248 tablet_copy_service.cc:433] P a5cb74a14a8d434e986ac7d988731e8a: ending tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-a68699dbcae34cfc94f2559e0516e5e8 on tablet a68699dbcae34cfc94f2559e0516e5e8 with peer 1fe6ac5a1583483eb93f50988c1b22a8
I20240502 14:15:45.193658 28464 tablet_bootstrap.cc:492] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=348 overwritten=0 applied=348 ignored=0} inserts{seen=1020 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:45.194043 28464 tablet_bootstrap.cc:492] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap complete.
I20240502 14:15:45.194332 28464 ts_tablet_manager.cc:1367] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent bootstrapping tablet: real 0.610s user 0.400s sys 0.025s
I20240502 14:15:45.194952 28464 raft_consensus.cc:348] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:45.195233 28464 raft_consensus.cc:729] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fe6ac5a1583483eb93f50988c1b22a8, State: Initialized, Role: LEARNER
I20240502 14:15:45.195430 28464 consensus_queue.cc:260] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 348, Last appended: 1.348, Last appended by leader: 348, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:45.196007 28464 ts_tablet_manager.cc:1398] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:45.196916 25248 tablet_copy_service.cc:342] P a5cb74a14a8d434e986ac7d988731e8a: Request end of tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-def417ebb8354201bb6c17eed13ebbf7 received from {username='slave'} at 127.21.58.72:34387
I20240502 14:15:45.197050 25248 tablet_copy_service.cc:433] P a5cb74a14a8d434e986ac7d988731e8a: ending tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-def417ebb8354201bb6c17eed13ebbf7 on tablet def417ebb8354201bb6c17eed13ebbf7 with peer 1fe6ac5a1583483eb93f50988c1b22a8
I20240502 14:15:45.275000 28467 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=463 overwritten=0 applied=463 ignored=0} inserts{seen=1145 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:45.275424 28467 tablet_bootstrap.cc:492] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: Bootstrap complete.
I20240502 14:15:45.275714 28467 ts_tablet_manager.cc:1367] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent bootstrapping tablet: real 0.679s user 0.479s sys 0.041s
I20240502 14:15:45.276343 28467 raft_consensus.cc:348] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:45.276526 28467 raft_consensus.cc:729] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fe6ac5a1583483eb93f50988c1b22a8, State: Initialized, Role: LEARNER
I20240502 14:15:45.276723 28467 consensus_queue.cc:260] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 463, Last appended: 1.463, Last appended by leader: 463, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: true } }
I20240502 14:15:45.277258 28467 ts_tablet_manager.cc:1398] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:45.278204 25509 tablet_copy_service.cc:342] P 1360ad293a3e4ecebf03dbf14cca9cb4: Request end of tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-60e291dc59934bc38239bb84b76975c9 received from {username='slave'} at 127.21.58.72:53975
I20240502 14:15:45.278362 25509 tablet_copy_service.cc:433] P 1360ad293a3e4ecebf03dbf14cca9cb4: ending tablet copy session 1fe6ac5a1583483eb93f50988c1b22a8-60e291dc59934bc38239bb84b76975c9 on tablet 60e291dc59934bc38239bb84b76975c9 with peer 1fe6ac5a1583483eb93f50988c1b22a8
I20240502 14:15:45.441542 28238 raft_consensus.cc:1203] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.403->[1.404-1.404] Dedup: 1.404->[]
I20240502 14:15:45.445235 28238 raft_consensus.cc:1203] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.462->[1.463-1.463] Dedup: 1.463->[]
I20240502 14:15:45.471489 28514 raft_consensus.cc:1050] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 to VOTER
I20240502 14:15:45.472105 28514 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391, Committed index: 391, Last appended: 1.391, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.474153 26939 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240502 14:15:45.474200 28238 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240502 14:15:45.474704 28379 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240502 14:15:45.475023 28512 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240502 14:15:45.477699 28379 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.392: config changed from index 391 to 392, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.478305 26939 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.481961 28238 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.483274 25082 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 391 to 392, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:45.499505 28238 raft_consensus.cc:1203] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.347->[1.348-1.348] Dedup: 1.348->[]
I20240502 14:15:45.574615 25488 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0723f00c11d744f8b244a3d003ce7b7a"
dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53406
I20240502 14:15:45.574824 25488 raft_consensus.cc:595] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:45.827452 25488 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0723f00c11d744f8b244a3d003ce7b7a"
dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53430
I20240502 14:15:45.827692 25488 raft_consensus.cc:595] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:45.858012 28549 raft_consensus.cc:979] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: : Instructing follower 103df70e837147d9a992da0e46ece12e to start an election
I20240502 14:15:45.858212 28549 raft_consensus.cc:1067] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Signalling peer 103df70e837147d9a992da0e46ece12e to start an election
I20240502 14:15:45.858927 26939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0723f00c11d744f8b244a3d003ce7b7a"
dest_uuid: "103df70e837147d9a992da0e46ece12e"
from {username='slave'} at 127.21.58.67:56921
I20240502 14:15:45.859200 26939 raft_consensus.cc:482] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:45.859349 26939 raft_consensus.cc:3037] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:45.861456 26939 raft_consensus.cc:504] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.862975 26939 leader_election.cc:288] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [CANDIDATE]: Term 2 election: Requested vote from peers 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255), 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72:36113)
I20240502 14:15:45.863423 25488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723f00c11d744f8b244a3d003ce7b7a" candidate_uuid: "103df70e837147d9a992da0e46ece12e" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: true dest_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4"
I20240502 14:15:45.863656 25488 raft_consensus.cc:3032] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:45.863756 25488 raft_consensus.cc:729] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1360ad293a3e4ecebf03dbf14cca9cb4, State: Running, Role: LEADER
I20240502 14:15:45.864033 25488 consensus_queue.cc:260] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 1.392, Last appended by leader: 392, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.864413 25488 raft_consensus.cc:3037] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:45.866863 25488 raft_consensus.cc:2454] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103df70e837147d9a992da0e46ece12e in term 2.
I20240502 14:15:45.867417 26873 leader_election.cc:302] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103df70e837147d9a992da0e46ece12e, 1360ad293a3e4ecebf03dbf14cca9cb4; no voters:
I20240502 14:15:45.867869 28550 raft_consensus.cc:2781] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:45.868389 28550 raft_consensus.cc:686] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 2 LEADER]: Becoming Leader. State: Replica: 103df70e837147d9a992da0e46ece12e, State: Running, Role: LEADER
I20240502 14:15:45.868932 28550 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 1.392, Last appended by leader: 392, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.869980 28238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723f00c11d744f8b244a3d003ce7b7a" candidate_uuid: "103df70e837147d9a992da0e46ece12e" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: true dest_uuid: "1fe6ac5a1583483eb93f50988c1b22a8"
I20240502 14:15:45.870270 28238 raft_consensus.cc:3037] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:45.872278 25084 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e reported cstate change: term changed from 1 to 2, leader changed from 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) to 103df70e837147d9a992da0e46ece12e (127.21.58.71). New cstate: current_term: 2 leader_uuid: "103df70e837147d9a992da0e46ece12e" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:45.872568 28238 raft_consensus.cc:2454] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103df70e837147d9a992da0e46ece12e in term 2.
I20240502 14:15:45.888906 28549 raft_consensus.cc:1050] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4: attempting to promote NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 to VOTER
I20240502 14:15:45.889474 28549 consensus_queue.cc:237] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 463, Committed index: 463, Last appended: 1.463, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 464 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.891662 28237 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 463. Preceding OpId from leader: term: 1 index: 464. (index mismatch)
I20240502 14:15:45.891695 26674 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 463. Preceding OpId from leader: term: 1 index: 464. (index mismatch)
I20240502 14:15:45.892236 28514 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 464, Last known committed idx: 463, Time since last communication: 0.000s
I20240502 14:15:45.892980 28545 raft_consensus.cc:1050] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a: attempting to promote NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 to VOTER
I20240502 14:15:45.893260 28514 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 464, Last known committed idx: 463, Time since last communication: 0.000s
I20240502 14:15:45.893730 28545 consensus_queue.cc:237] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.895344 28549 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.464: config changed from index 463 to 464, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 464 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.895974 28238 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Committing config change with OpId 1.464: config changed from index 463 to 464, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 464 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.896734 28237 raft_consensus.cc:1261] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240502 14:15:45.897404 25357 raft_consensus.cc:1261] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240502 14:15:45.897980 28544 consensus_queue.cc:1035] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240502 14:15:45.898372 28498 consensus_queue.cc:1035] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240502 14:15:45.900710 25084 catalog_manager.cc:5484] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 463 to 464, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 464 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:45.901890 26674 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Committing config change with OpId 1.464: config changed from index 463 to 464, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 464 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "2bf9b3d134bc4ac79a05d942fc2fb68b" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 44755 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.905205 25488 consensus_queue.cc:237] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 464, Committed index: 464, Last appended: 1.464, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 465 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.905263 28498 raft_consensus.cc:2932] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index 348 to 349, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.907292 28237 raft_consensus.cc:1261] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Refusing update from remote peer 1360ad293a3e4ecebf03dbf14cca9cb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 464. Preceding OpId from leader: term: 1 index: 465. (index mismatch)
I20240502 14:15:45.907924 25359 raft_consensus.cc:2932] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 348 to 349, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.908702 28379 consensus_queue.cc:1035] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 465, Last known committed idx: 464, Time since last communication: 0.000s
I20240502 14:15:45.910452 28237 raft_consensus.cc:2932] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 348 to 349, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.910843 25081 catalog_manager.cc:5484] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 348 to 349, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c91321c0a57d4bec8eb3bfba60cee06e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:45.914821 25228 consensus_queue.cc:237] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.916376 28549 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 1 LEADER]: Committing config change with OpId 1.465: config changed from index 464 to 465, VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) evicted. New config: { opid_index: 465 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.917029 28237 raft_consensus.cc:2932] T 60e291dc59934bc38239bb84b76975c9 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Committing config change with OpId 1.465: config changed from index 464 to 465, VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) evicted. New config: { opid_index: 465 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.918118 28238 raft_consensus.cc:1261] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240502 14:15:45.918756 28545 consensus_queue.cc:1035] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240502 14:15:45.919922 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60e291dc59934bc38239bb84b76975c9 with cas_config_opid_index 464: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:45.921265 28545 raft_consensus.cc:2932] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.921075 25081 catalog_manager.cc:5484] T 60e291dc59934bc38239bb84b76975c9 P 1360ad293a3e4ecebf03dbf14cca9cb4 reported cstate change: config changed from index 464 to 465, VOTER 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69) evicted. New cstate: current_term: 1 leader_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" committed_config { opid_index: 465 OBSOLETE_local: false peers { permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:45.924531 25072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet def417ebb8354201bb6c17eed13ebbf7 with cas_config_opid_index 349: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:45.924928 28237 raft_consensus.cc:2932] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.925485 26657 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e291dc59934bc38239bb84b76975c9 with delete_type TABLET_DATA_TOMBSTONED (TS 2bf9b3d134bc4ac79a05d942fc2fb68b not found in new config with opid_index 465) from {username='slave'} at 127.0.0.1:52796
I20240502 14:15:45.926529 28617 tablet_replica.cc:330] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: stopping tablet replica
I20240502 14:15:45.927341 25084 catalog_manager.cc:5484] T def417ebb8354201bb6c17eed13ebbf7 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 349 to 350, VOTER c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:45.928622 28617 raft_consensus.cc:2229] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:45.928723 28545 raft_consensus.cc:1050] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: attempting to promote NON_VOTER 1fe6ac5a1583483eb93f50988c1b22a8 to VOTER
I20240502 14:15:45.929068 28617 raft_consensus.cc:2258] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:45.930218 28545 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 1.404, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:45.931483 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet def417ebb8354201bb6c17eed13ebbf7 with delete_type TABLET_DATA_TOMBSTONED (TS c91321c0a57d4bec8eb3bfba60cee06e not found in new config with opid_index 350) from {username='slave'} at 127.0.0.1:43744
I20240502 14:15:45.932090 28617 ts_tablet_manager.cc:1875] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:45.932677 28619 tablet_replica.cc:330] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: stopping tablet replica
I20240502 14:15:45.932906 28619 raft_consensus.cc:2229] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:45.933018 26544 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 404. Preceding OpId from leader: term: 1 index: 405. (index mismatch)
I20240502 14:15:45.933215 28619 raft_consensus.cc:2258] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:45.933694 28544 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I20240502 14:15:45.935323 28619 ts_tablet_manager.cc:1875] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:45.939437 28237 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 LEARNER]: Refusing update from remote peer a5cb74a14a8d434e986ac7d988731e8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 404. Preceding OpId from leader: term: 1 index: 405. (index mismatch)
I20240502 14:15:45.939560 28617 ts_tablet_manager.cc:1888] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.464
I20240502 14:15:45.939828 28617 log.cc:1196] T 60e291dc59934bc38239bb84b76975c9 P 2bf9b3d134bc4ac79a05d942fc2fb68b: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/wals/60e291dc59934bc38239bb84b76975c9
I20240502 14:15:45.940119 28545 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I20240502 14:15:45.940284 28619 ts_tablet_manager.cc:1888] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.349
I20240502 14:15:45.940459 28619 log.cc:1196] T def417ebb8354201bb6c17eed13ebbf7 P c91321c0a57d4bec8eb3bfba60cee06e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/def417ebb8354201bb6c17eed13ebbf7
I20240502 14:15:45.941160 25074 catalog_manager.cc:4830] TS 2bf9b3d134bc4ac79a05d942fc2fb68b (127.21.58.69:44755): tablet 60e291dc59934bc38239bb84b76975c9 (table rebalance_test_table_0 [id=eaec7bde71514b33a34da983e27530cb]) successfully deleted
I20240502 14:15:45.941371 25071 catalog_manager.cc:4830] TS c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): tablet def417ebb8354201bb6c17eed13ebbf7 (table rebalance_test_table_1 [id=4fb6232575984073bc329979710e2fa8]) successfully deleted
I20240502 14:15:45.943013 28498 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Committing config change with OpId 1.405: config changed from index 404 to 405, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.943917 26544 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Committing config change with OpId 1.405: config changed from index 404 to 405, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.943958 28238 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Committing config change with OpId 1.405: config changed from index 404 to 405, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:45.947228 25084 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a reported cstate change: config changed from index 404 to 405, 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5cb74a14a8d434e986ac7d988731e8a" committed_config { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:46.065323 25228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8"
dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38700
I20240502 14:15:46.065536 25228 raft_consensus.cc:595] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:46.102052 25228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8"
dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38734
I20240502 14:15:46.102234 25228 raft_consensus.cc:595] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:46.261369 28238 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 FOLLOWER]: Refusing update from remote peer 103df70e837147d9a992da0e46ece12e: Log matching property violated. Preceding OpId in replica: term: 1 index: 392. Preceding OpId from leader: term: 2 index: 393. (index mismatch)
I20240502 14:15:46.262037 28550 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I20240502 14:15:46.265769 25488 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Refusing update from remote peer 103df70e837147d9a992da0e46ece12e: Log matching property violated. Preceding OpId in replica: term: 1 index: 392. Preceding OpId from leader: term: 2 index: 393. (index mismatch)
I20240502 14:15:46.266582 28610 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1360ad293a3e4ecebf03dbf14cca9cb4" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 33255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I20240502 14:15:46.277297 26939 consensus_queue.cc:237] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 393, Committed index: 393, Last appended: 2.393, Last appended by leader: 392, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:46.278905 28238 raft_consensus.cc:1261] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 FOLLOWER]: Refusing update from remote peer 103df70e837147d9a992da0e46ece12e: Log matching property violated. Preceding OpId in replica: term: 2 index: 393. Preceding OpId from leader: term: 2 index: 394. (index mismatch)
I20240502 14:15:46.279389 28610 consensus_queue.cc:1035] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 394, Last known committed idx: 393, Time since last communication: 0.000s
I20240502 14:15:46.281472 28550 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e [term 2 LEADER]: Committing config change with OpId 2.394: config changed from index 392 to 394, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:46.282132 28238 raft_consensus.cc:2932] T 0723f00c11d744f8b244a3d003ce7b7a P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 FOLLOWER]: Committing config change with OpId 2.394: config changed from index 392 to 394, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:46.285271 25071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0723f00c11d744f8b244a3d003ce7b7a with cas_config_opid_index 392: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:46.286742 25081 catalog_manager.cc:5484] T 0723f00c11d744f8b244a3d003ce7b7a P 103df70e837147d9a992da0e46ece12e reported cstate change: config changed from index 392 to 394, VOTER 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67) evicted. New cstate: current_term: 2 leader_uuid: "103df70e837147d9a992da0e46ece12e" committed_config { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "103df70e837147d9a992da0e46ece12e" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 37231 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:46.290112 25469 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723f00c11d744f8b244a3d003ce7b7a with delete_type TABLET_DATA_TOMBSTONED (TS 1360ad293a3e4ecebf03dbf14cca9cb4 not found in new config with opid_index 394) from {username='slave'} at 127.0.0.1:40350
I20240502 14:15:46.290935 28667 tablet_replica.cc:330] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: stopping tablet replica
I20240502 14:15:46.291149 28667 raft_consensus.cc:2229] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:46.291332 28667 raft_consensus.cc:2258] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:46.293159 28667 ts_tablet_manager.cc:1875] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:46.299412 28667 ts_tablet_manager.cc:1888] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.393
I20240502 14:15:46.299527 28667 log.cc:1196] T 0723f00c11d744f8b244a3d003ce7b7a P 1360ad293a3e4ecebf03dbf14cca9cb4: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:46.300392 25072 catalog_manager.cc:4830] TS 1360ad293a3e4ecebf03dbf14cca9cb4 (127.21.58.67:33255): tablet 0723f00c11d744f8b244a3d003ce7b7a (table rebalance_test_table_3 [id=c27f89dbb8084518b0a53aa80d1ebe66]) successfully deleted
I20240502 14:15:46.328302 25228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8"
dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38752
I20240502 14:15:46.328490 25228 raft_consensus.cc:595] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:46.353538 25228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8"
dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38764
I20240502 14:15:46.353715 25228 raft_consensus.cc:595] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:46.364101 28545 raft_consensus.cc:979] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: : Instructing follower 1fe6ac5a1583483eb93f50988c1b22a8 to start an election
I20240502 14:15:46.364270 28545 raft_consensus.cc:1067] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Signalling peer 1fe6ac5a1583483eb93f50988c1b22a8 to start an election
I20240502 14:15:46.364874 28238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8"
dest_uuid: "1fe6ac5a1583483eb93f50988c1b22a8"
from {username='slave'} at 127.21.58.65:44743
I20240502 14:15:46.365139 28238 raft_consensus.cc:482] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:46.365267 28238 raft_consensus.cc:3037] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:46.367345 28238 raft_consensus.cc:504] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:46.368880 28238 leader_election.cc:288] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [CANDIDATE]: Term 2 election: Requested vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045), c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117)
I20240502 14:15:46.369184 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8" candidate_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 405 } } ignore_live_leader: true dest_uuid: "a5cb74a14a8d434e986ac7d988731e8a"
I20240502 14:15:46.369429 25228 raft_consensus.cc:3032] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:46.369553 25228 raft_consensus.cc:729] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5cb74a14a8d434e986ac7d988731e8a, State: Running, Role: LEADER
I20240502 14:15:46.369872 25228 consensus_queue.cc:260] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 405, Committed index: 405, Last appended: 1.405, Last appended by leader: 405, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:46.370385 25228 raft_consensus.cc:3037] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:46.374076 25228 raft_consensus.cc:2454] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe6ac5a1583483eb93f50988c1b22a8 in term 2.
I20240502 14:15:46.374845 28172 leader_election.cc:302] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe6ac5a1583483eb93f50988c1b22a8, a5cb74a14a8d434e986ac7d988731e8a; no voters:
I20240502 14:15:46.375613 28508 raft_consensus.cc:2781] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:46.376114 28508 raft_consensus.cc:686] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 LEADER]: Becoming Leader. State: Replica: 1fe6ac5a1583483eb93f50988c1b22a8, State: Running, Role: LEADER
I20240502 14:15:46.376740 28508 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 405, Committed index: 405, Last appended: 1.405, Last appended by leader: 405, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:46.380515 25081 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 reported cstate change: term changed from 1 to 2, leader changed from a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) to 1fe6ac5a1583483eb93f50988c1b22a8 (127.21.58.72). New cstate: current_term: 2 leader_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" committed_config { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:46.383586 26544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68699dbcae34cfc94f2559e0516e5e8" candidate_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 405 } } ignore_live_leader: true dest_uuid: "c69e4233b679437b8eb3b811680fa196"
I20240502 14:15:46.383836 26544 raft_consensus.cc:3037] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:46.386586 26544 raft_consensus.cc:2454] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe6ac5a1583483eb93f50988c1b22a8 in term 2.
I20240502 14:15:46.848977 26544 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Refusing update from remote peer 1fe6ac5a1583483eb93f50988c1b22a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 2 index: 406. (index mismatch)
I20240502 14:15:46.849577 28508 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20240502 14:15:46.854306 25228 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Refusing update from remote peer 1fe6ac5a1583483eb93f50988c1b22a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 2 index: 406. (index mismatch)
I20240502 14:15:46.855023 28508 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20240502 14:15:46.862900 28238 consensus_queue.cc:237] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 406, Committed index: 406, Last appended: 2.406, Last appended by leader: 405, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:46.864221 26544 raft_consensus.cc:1261] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Refusing update from remote peer 1fe6ac5a1583483eb93f50988c1b22a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 406. Preceding OpId from leader: term: 2 index: 407. (index mismatch)
I20240502 14:15:46.864674 28508 consensus_queue.cc:1035] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20240502 14:15:46.866551 28696 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 2 LEADER]: Committing config change with OpId 2.407: config changed from index 405 to 407, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:46.867121 26544 raft_consensus.cc:2932] T a68699dbcae34cfc94f2559e0516e5e8 P c69e4233b679437b8eb3b811680fa196 [term 2 FOLLOWER]: Committing config change with OpId 2.407: config changed from index 405 to 407, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } } }
I20240502 14:15:46.870724 25074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a68699dbcae34cfc94f2559e0516e5e8 with cas_config_opid_index 405: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:46.871366 25081 catalog_manager.cc:5484] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 reported cstate change: config changed from index 405 to 407, VOTER a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" committed_config { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "c69e4233b679437b8eb3b811680fa196" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:46.874498 25207 tablet_service.cc:1514] Processing DeleteTablet for tablet a68699dbcae34cfc94f2559e0516e5e8 with delete_type TABLET_DATA_TOMBSTONED (TS a5cb74a14a8d434e986ac7d988731e8a not found in new config with opid_index 407) from {username='slave'} at 127.0.0.1:34208
I20240502 14:15:46.875205 28713 tablet_replica.cc:330] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: stopping tablet replica
I20240502 14:15:46.875430 28713 raft_consensus.cc:2229] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:46.875727 28713 raft_consensus.cc:2258] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:46.877321 28713 ts_tablet_manager.cc:1875] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:46.881587 28713 ts_tablet_manager.cc:1888] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.406
I20240502 14:15:46.881685 28713 log.cc:1196] T a68699dbcae34cfc94f2559e0516e5e8 P a5cb74a14a8d434e986ac7d988731e8a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:46.882462 25072 catalog_manager.cc:4830] TS a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): tablet a68699dbcae34cfc94f2559e0516e5e8 (table rebalance_test_table_2 [id=b9d2e585423c419ab821f0467fb014d3]) successfully deleted
I20240502 14:15:47.126673 28218 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.129791 25469 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:47.133014 26921 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.154628 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.169653 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.170275 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:47.184759 25207 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.211374 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.370736 26921 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.381443 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.391786 28218 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.393543 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.409061 25207 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.411640 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:15:47.418932 25469 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:47.419898 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.750907 26921 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.762847 26657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.771178 26787 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.789913 25469 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:47.789975 28218 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:15:47.814713 25207 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.815622 26526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:47.829931 25339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9ab9bceb14584c9b9819e55247c54100 | 127.21.58.126:40155 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:45959 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
103df70e837147d9a992da0e46ece12e | 127.21.58.71:37231 | HEALTHY | <none> | 1 | 0
1360ad293a3e4ecebf03dbf14cca9cb4 | 127.21.58.67:33255 | HEALTHY | <none> | 4 | 0
1fe6ac5a1583483eb93f50988c1b22a8 | 127.21.58.72:36113 | HEALTHY | <none> | 1 | 0
2bf9b3d134bc4ac79a05d942fc2fb68b | 127.21.58.69:44755 | HEALTHY | <none> | 2 | 0
6365836c0f2b428bbeb4d69b9ceda2fe | 127.21.58.70:35761 | HEALTHY | <none> | 1 | 0
a5cb74a14a8d434e986ac7d988731e8a | 127.21.58.65:46045 | HEALTHY | <none> | 2 | 0
c69e4233b679437b8eb3b811680fa196 | 127.21.58.68:43117 | HEALTHY | <none> | 2 | 0
c91321c0a57d4bec8eb3bfba60cee06e | 127.21.58.66:44323 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.21.58.65 | experimental | 127.21.58.65:46045
local_ip_for_outbound_sockets | 127.21.58.66 | experimental | 127.21.58.66:44323
local_ip_for_outbound_sockets | 127.21.58.67 | experimental | 127.21.58.67:33255
local_ip_for_outbound_sockets | 127.21.58.68 | experimental | 127.21.58.68:43117
local_ip_for_outbound_sockets | 127.21.58.69 | experimental | 127.21.58.69:44755
local_ip_for_outbound_sockets | 127.21.58.70 | experimental | 127.21.58.70:35761
local_ip_for_outbound_sockets | 127.21.58.71 | experimental | 127.21.58.71:37231
local_ip_for_outbound_sockets | 127.21.58.72 | experimental | 127.21.58.72:36113
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.58.65:46045
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.58.66:44323
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.58.67:33255
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.21.58.68:43117
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.21.58.69:44755
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.21.58.70:35761
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.21.58.71:37231
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.21.58.72:36113
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:45959 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:15:47.896385 21737 log_verifier.cc:126] Checking tablet 02f1d29d09964aa3bfb918e985c18340
I20240502 14:15:48.060986 21737 log_verifier.cc:177] Verified matching terms for 436 ops in tablet 02f1d29d09964aa3bfb918e985c18340
I20240502 14:15:48.061551 21737 log_verifier.cc:126] Checking tablet 0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:48.208130 21737 log_verifier.cc:177] Verified matching terms for 394 ops in tablet 0723f00c11d744f8b244a3d003ce7b7a
I20240502 14:15:48.208623 21737 log_verifier.cc:126] Checking tablet 232f776da8074d5bb2643f4add1cb06a
I20240502 14:15:48.360105 21737 log_verifier.cc:177] Verified matching terms for 403 ops in tablet 232f776da8074d5bb2643f4add1cb06a
I20240502 14:15:48.360610 21737 log_verifier.cc:126] Checking tablet 291df3cd98de4152ab6aaf069bc45a36
I20240502 14:15:48.494242 21737 log_verifier.cc:177] Verified matching terms for 398 ops in tablet 291df3cd98de4152ab6aaf069bc45a36
I20240502 14:15:48.494731 21737 log_verifier.cc:126] Checking tablet 37ae6a3602fb4437a0e7c337494ea8e8
I20240502 14:15:48.649875 21737 log_verifier.cc:177] Verified matching terms for 426 ops in tablet 37ae6a3602fb4437a0e7c337494ea8e8
I20240502 14:15:48.650383 21737 log_verifier.cc:126] Checking tablet 503e0727a7a14863847f375032cb8144
I20240502 14:15:48.801786 21737 log_verifier.cc:177] Verified matching terms for 416 ops in tablet 503e0727a7a14863847f375032cb8144
I20240502 14:15:48.802297 21737 log_verifier.cc:126] Checking tablet 537b91a6ab124022a1cdfb077055d06f
I20240502 14:15:48.958159 21737 log_verifier.cc:177] Verified matching terms for 412 ops in tablet 537b91a6ab124022a1cdfb077055d06f
I20240502 14:15:48.958665 21737 log_verifier.cc:126] Checking tablet 57558d7d2f9943dc8fe5fd127f3062e8
I20240502 14:15:49.106544 21737 log_verifier.cc:177] Verified matching terms for 401 ops in tablet 57558d7d2f9943dc8fe5fd127f3062e8
I20240502 14:15:49.107054 21737 log_verifier.cc:126] Checking tablet 60e291dc59934bc38239bb84b76975c9
I20240502 14:15:49.262579 21737 log_verifier.cc:177] Verified matching terms for 465 ops in tablet 60e291dc59934bc38239bb84b76975c9
I20240502 14:15:49.263115 21737 log_verifier.cc:126] Checking tablet 66c85e449bea4f829fe6d1c79206591d
I20240502 14:15:49.400041 21737 log_verifier.cc:177] Verified matching terms for 384 ops in tablet 66c85e449bea4f829fe6d1c79206591d
I20240502 14:15:49.400497 21737 log_verifier.cc:126] Checking tablet a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:49.561544 21737 log_verifier.cc:177] Verified matching terms for 407 ops in tablet a68699dbcae34cfc94f2559e0516e5e8
I20240502 14:15:49.562003 21737 log_verifier.cc:126] Checking tablet de14694a11e14f6082bba7fa70f51553
I20240502 14:15:49.577195 21737 log_verifier.cc:177] Verified matching terms for 5 ops in tablet de14694a11e14f6082bba7fa70f51553
I20240502 14:15:49.577296 21737 log_verifier.cc:126] Checking tablet def417ebb8354201bb6c17eed13ebbf7
I20240502 14:15:49.734566 21737 log_verifier.cc:177] Verified matching terms for 350 ops in tablet def417ebb8354201bb6c17eed13ebbf7
I20240502 14:15:49.734968 21737 log_verifier.cc:126] Checking tablet e30b47ad42124444aadeabc9924e8681
I20240502 14:15:49.866847 21737 log_verifier.cc:177] Verified matching terms for 393 ops in tablet e30b47ad42124444aadeabc9924e8681
I20240502 14:15:49.867422 21737 log_verifier.cc:126] Checking tablet ead87caf4f9c4695a9c1b8b64852cabf
I20240502 14:15:50.021055 21737 log_verifier.cc:177] Verified matching terms for 414 ops in tablet ead87caf4f9c4695a9c1b8b64852cabf
I20240502 14:15:50.021589 21737 log_verifier.cc:126] Checking tablet fcc673d4727a4c049228070d9e2ada05
I20240502 14:15:50.196074 21737 log_verifier.cc:177] Verified matching terms for 415 ops in tablet fcc673d4727a4c049228070d9e2ada05
I20240502 14:15:50.202839 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25146
I20240502 14:15:50.204089 25268 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:50.443506 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25146
W20240502 14:15:50.485587 26740 connection.cc:535] server connection from 127.21.58.65:32785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:50.486359 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25277
I20240502 14:15:50.487514 25398 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:15:50.673564 26478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.65:46045: connect: Connection refused (error 111)
W20240502 14:15:50.675128 26478 consensus_peers.cc:485] T 02f1d29d09964aa3bfb918e985c18340 P c69e4233b679437b8eb3b811680fa196 -> Peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): Couldn't send request to peer a5cb74a14a8d434e986ac7d988731e8a. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:46045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:50.731066 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25277
W20240502 14:15:50.767992 26875 connection.cc:535] server connection from 127.21.58.66:36071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:50.767989 26612 connection.cc:535] server connection from 127.21.58.66:49711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:50.768812 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25407
I20240502 14:15:50.769840 25529 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:15:50.843315 26610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.66:44323: connect: Connection refused (error 111)
W20240502 14:15:50.844894 26610 consensus_peers.cc:485] T ead87caf4f9c4695a9c1b8b64852cabf P 2bf9b3d134bc4ac79a05d942fc2fb68b -> Peer c91321c0a57d4bec8eb3bfba60cee06e (127.21.58.66:44323): Couldn't send request to peer c91321c0a57d4bec8eb3bfba60cee06e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:44323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:51.010623 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25407
I20240502 14:15:51.057637 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 26462
I20240502 14:15:51.064285 26585 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:51.233810 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 26462
W20240502 14:15:51.264261 28174 connection.cc:535] client connection to 127.21.58.68:43117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:51.264506 28174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:51.264531 26874 connection.cc:535] server connection from 127.21.58.68:50775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:51.265053 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 26595
W20240502 14:15:51.266641 28174 consensus_peers.cc:485] T a68699dbcae34cfc94f2559e0516e5e8 P 1fe6ac5a1583483eb93f50988c1b22a8 -> Peer c69e4233b679437b8eb3b811680fa196 (127.21.58.68:43117): Couldn't send request to peer c69e4233b679437b8eb3b811680fa196. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.68:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:51.271384 26716 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:51.455006 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 26595
I20240502 14:15:51.488054 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 26725
I20240502 14:15:51.493724 26847 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:51.545163 28859 raft_consensus.cc:482] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a5cb74a14a8d434e986ac7d988731e8a)
I20240502 14:15:51.545411 28859 raft_consensus.cc:504] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "a5cb74a14a8d434e986ac7d988731e8a" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46045 } } peers { permanent_uuid: "1fe6ac5a1583483eb93f50988c1b22a8" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36113 } attrs { promote: false } }
I20240502 14:15:51.546029 28859 leader_election.cc:288] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045)
W20240502 14:15:51.547510 28172 leader_election.cc:334] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a5cb74a14a8d434e986ac7d988731e8a (127.21.58.65:46045): Network error: Client connection negotiation failed: client connection to 127.21.58.65:46045: connect: Connection refused (error 111)
I20240502 14:15:51.547672 28172 leader_election.cc:302] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1fe6ac5a1583483eb93f50988c1b22a8; no voters: a5cb74a14a8d434e986ac7d988731e8a
I20240502 14:15:51.547942 28859 raft_consensus.cc:2726] T def417ebb8354201bb6c17eed13ebbf7 P 1fe6ac5a1583483eb93f50988c1b22a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:51.656657 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 26725
I20240502 14:15:51.686576 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 26855
I20240502 14:15:51.692298 26982 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:51.845960 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 26855
I20240502 14:15:51.874585 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 28156
I20240502 14:15:51.880329 28278 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:52.006672 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 28156
I20240502 14:15:52.028591 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 25056
I20240502 14:15:52.029597 25117 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:52.193850 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 25056
2024-05-02T14:15:52Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (49279 ms)
[----------] 1 test from RebalanceParamTest (49279 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240502 14:15:52.289310 21737 test_util.cc:268] Using random seed: 1033543697
I20240502 14:15:52.291047 21737 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:15:52.291389 21737 ts_itest-base.cc:113] --------------
I20240502 14:15:52.291426 21737 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:15:52.291473 21737 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:15:52.291503 21737 ts_itest-base.cc:116] --------------
2024-05-02T14:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:15:52Z Disabled control of system clock
I20240502 14:15:52.347800 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:42921
--webserver_interface=127.21.58.126
--webserver_port=0
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.58.126:42921
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:15:52.691067 28884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:52.691416 28884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:52.691548 28884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:52.699460 28884 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:52.699575 28884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:15:52.699666 28884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:52.699735 28884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:15:52.699797 28884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:15:52.708671 28884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.58.126:42921
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.58.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:52.710000 28884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:52.711802 28884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:52.722491 28889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:52.723201 28890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:52.723778 28892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:52.724413 28884 server_base.cc:1034] running on GCE node
I20240502 14:15:52.725059 28884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:52.726342 28884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:52.727526 28884 hybrid_clock.cc:648] HybridClock initialized: now 1714659352727481 us; error 38 us; skew 500 ppm
I20240502 14:15:52.727849 28884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:52.729851 28884 webserver.cc:460] Webserver started at http://127.21.58.126:35847/ using document root <none> and password file <none>
I20240502 14:15:52.730484 28884 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:52.730590 28884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:52.730908 28884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:52.733477 28884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "392d24eabd00494bba32271b023efb45"
format_stamp: "Formatted at 2024-05-02 14:15:52 on dist-test-slave-wtgm"
I20240502 14:15:52.734238 28884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "392d24eabd00494bba32271b023efb45"
format_stamp: "Formatted at 2024-05-02 14:15:52 on dist-test-slave-wtgm"
I20240502 14:15:52.739023 28884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 14:15:52.742161 28898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:52.743461 28884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:52.743634 28884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
uuid: "392d24eabd00494bba32271b023efb45"
format_stamp: "Formatted at 2024-05-02 14:15:52 on dist-test-slave-wtgm"
I20240502 14:15:52.743822 28884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:52.760759 28884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:52.761529 28884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:52.761802 28884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:52.779983 28884 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.126:42921
I20240502 14:15:52.779991 28949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.126:42921 every 8 connection(s)
I20240502 14:15:52.781265 28884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:15:52.784757 28951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:52.786696 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 28884
I20240502 14:15:52.786872 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:15:52.794799 28951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45: Bootstrap starting.
I20240502 14:15:52.797797 28951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:52.799108 28951 log.cc:824] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:52.801599 28951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45: No bootstrap required, opened a new log
I20240502 14:15:52.807603 28951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392d24eabd00494bba32271b023efb45" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42921 } }
I20240502 14:15:52.807900 28951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:52.807978 28951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392d24eabd00494bba32271b023efb45, State: Initialized, Role: FOLLOWER
I20240502 14:15:52.808481 28951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392d24eabd00494bba32271b023efb45" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42921 } }
I20240502 14:15:52.808662 28951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:15:52.808760 28951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:15:52.808869 28951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:52.810786 28951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392d24eabd00494bba32271b023efb45" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42921 } }
I20240502 14:15:52.811221 28951 leader_election.cc:302] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 392d24eabd00494bba32271b023efb45; no voters:
I20240502 14:15:52.811724 28951 leader_election.cc:288] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:15:52.811856 28956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:52.812868 28956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [term 1 LEADER]: Becoming Leader. State: Replica: 392d24eabd00494bba32271b023efb45, State: Running, Role: LEADER
I20240502 14:15:52.813483 28956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392d24eabd00494bba32271b023efb45" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42921 } }
I20240502 14:15:52.813652 28951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:15:52.816459 28957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "392d24eabd00494bba32271b023efb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392d24eabd00494bba32271b023efb45" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42921 } } }
I20240502 14:15:52.816534 28958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 392d24eabd00494bba32271b023efb45. Latest consensus state: current_term: 1 leader_uuid: "392d24eabd00494bba32271b023efb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392d24eabd00494bba32271b023efb45" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42921 } } }
I20240502 14:15:52.816759 28957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:52.816838 28958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:52.818048 28964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:15:52.823586 28964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:15:52.832581 28964 catalog_manager.cc:1347] Generated new cluster ID: a4c44222b01d49f5b088127dba409d00
I20240502 14:15:52.832682 28964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:15:52.847553 28964 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:15:52.848831 28964 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:15:52.859450 28964 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 392d24eabd00494bba32271b023efb45: Generated new TSK 0
I20240502 14:15:52.860224 28964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:15:52.869645 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.65:0
--local_ip_for_outbound_sockets=127.21.58.65
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:15:53.229380 28975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:53.229641 28975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:53.229769 28975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:53.237337 28975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:53.237452 28975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:53.237632 28975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.65
I20240502 14:15:53.246343 28975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.65
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:53.247740 28975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:53.249568 28975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:53.261117 28980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.261848 28981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.262466 28983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:53.263240 28975 server_base.cc:1034] running on GCE node
I20240502 14:15:53.263819 28975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:53.264714 28975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:53.265951 28975 hybrid_clock.cc:648] HybridClock initialized: now 1714659353265898 us; error 44 us; skew 500 ppm
I20240502 14:15:53.266222 28975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:53.268376 28975 webserver.cc:460] Webserver started at http://127.21.58.65:44671/ using document root <none> and password file <none>
I20240502 14:15:53.269050 28975 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:53.269142 28975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:53.269414 28975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:53.271955 28975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9329e8a7a2d1453d90b76b5965315837"
format_stamp: "Formatted at 2024-05-02 14:15:53 on dist-test-slave-wtgm"
I20240502 14:15:53.272651 28975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9329e8a7a2d1453d90b76b5965315837"
format_stamp: "Formatted at 2024-05-02 14:15:53 on dist-test-slave-wtgm"
I20240502 14:15:53.277438 28975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:15:53.280674 28989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.281886 28975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:53.282044 28975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9329e8a7a2d1453d90b76b5965315837"
format_stamp: "Formatted at 2024-05-02 14:15:53 on dist-test-slave-wtgm"
I20240502 14:15:53.282230 28975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:53.298934 28975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:53.299958 28975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:53.300261 28975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:53.301446 28975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:53.303458 28975 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:53.303570 28975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.303684 28975 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:53.303763 28975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.334946 28975 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.65:46783
I20240502 14:15:53.334992 29101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.65:46783 every 8 connection(s)
I20240502 14:15:53.336241 28975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:15:53.338874 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 28975
I20240502 14:15:53.339135 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:15:53.342864 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.66:0
--local_ip_for_outbound_sockets=127.21.58.66
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:53.349905 29102 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:53.350214 29102 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:53.351277 29102 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:53.353482 28915 ts_manager.cc:194] Registered new tserver with Master: 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:15:53.355387 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.65:32829
W20240502 14:15:53.677080 29106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:53.677356 29106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:53.677443 29106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:53.684679 29106 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:53.684798 29106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:53.684921 29106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.66
I20240502 14:15:53.694221 29106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.66
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:53.695537 29106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:53.697196 29106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:53.708925 29111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.709546 29114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.710160 29112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:53.710964 29106 server_base.cc:1034] running on GCE node
I20240502 14:15:53.711510 29106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:53.712355 29106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:53.713544 29106 hybrid_clock.cc:648] HybridClock initialized: now 1714659353713494 us; error 42 us; skew 500 ppm
I20240502 14:15:53.713886 29106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:53.716087 29106 webserver.cc:460] Webserver started at http://127.21.58.66:35087/ using document root <none> and password file <none>
I20240502 14:15:53.716761 29106 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:53.716861 29106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:53.717165 29106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:53.719738 29106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dbabfd51061b4e509d907a3622febd2e"
format_stamp: "Formatted at 2024-05-02 14:15:53 on dist-test-slave-wtgm"
I20240502 14:15:53.720458 29106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dbabfd51061b4e509d907a3622febd2e"
format_stamp: "Formatted at 2024-05-02 14:15:53 on dist-test-slave-wtgm"
I20240502 14:15:53.725410 29106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:15:53.728578 29120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.729807 29106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:53.729954 29106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dbabfd51061b4e509d907a3622febd2e"
format_stamp: "Formatted at 2024-05-02 14:15:53 on dist-test-slave-wtgm"
I20240502 14:15:53.730144 29106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:53.752382 29106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:53.753222 29106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:53.753473 29106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:53.754405 29106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:53.756001 29106 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:53.756086 29106 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.756196 29106 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:53.756268 29106 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.790644 29106 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.66:44753
I20240502 14:15:53.790699 29234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.66:44753 every 8 connection(s)
I20240502 14:15:53.792009 29106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:15:53.796283 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 29106
I20240502 14:15:53.796454 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:15:53.799901 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:0
--local_ip_for_outbound_sockets=127.21.58.67
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:53.806053 29235 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:53.806370 29235 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:53.807439 29235 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:53.808961 28915 ts_manager.cc:194] Registered new tserver with Master: dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:15:53.809754 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.66:49133
W20240502 14:15:54.136224 29239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:54.136485 29239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:54.136571 29239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:54.143961 29239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:54.144069 29239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:54.144240 29239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:15:54.153399 29239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:54.154649 29239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:54.156327 29239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:54.168242 29245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:54.168882 29247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:54.169494 29244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:54.170276 29239 server_base.cc:1034] running on GCE node
I20240502 14:15:54.170851 29239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:54.171752 29239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:54.172950 29239 hybrid_clock.cc:648] HybridClock initialized: now 1714659354172907 us; error 32 us; skew 500 ppm
I20240502 14:15:54.173243 29239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:54.175380 29239 webserver.cc:460] Webserver started at http://127.21.58.67:41267/ using document root <none> and password file <none>
I20240502 14:15:54.176072 29239 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:54.176180 29239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:54.176499 29239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:54.178951 29239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1b3c20f7bfd548769f6bd40148590b74"
format_stamp: "Formatted at 2024-05-02 14:15:54 on dist-test-slave-wtgm"
I20240502 14:15:54.179734 29239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1b3c20f7bfd548769f6bd40148590b74"
format_stamp: "Formatted at 2024-05-02 14:15:54 on dist-test-slave-wtgm"
I20240502 14:15:54.184427 29239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 14:15:54.187713 29253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.188885 29239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:15:54.189040 29239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1b3c20f7bfd548769f6bd40148590b74"
format_stamp: "Formatted at 2024-05-02 14:15:54 on dist-test-slave-wtgm"
I20240502 14:15:54.189224 29239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:54.202483 29239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:54.203274 29239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:54.203528 29239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:54.204424 29239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:54.205801 29239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:54.205896 29239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.206009 29239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:54.206087 29239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.237617 29239 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:40039
I20240502 14:15:54.237671 29366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:40039 every 8 connection(s)
I20240502 14:15:54.238883 29239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:15:54.241179 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 29239
I20240502 14:15:54.241358 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:15:54.244833 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:0
--local_ip_for_outbound_sockets=127.21.58.68
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:54.253108 29367 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:54.253432 29367 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:54.254427 29367 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:54.255952 28915 ts_manager.cc:194] Registered new tserver with Master: 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:15:54.256737 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:57165
I20240502 14:15:54.358280 29102 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
W20240502 14:15:54.580274 29370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:54.580538 29370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:54.580685 29370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:54.588063 29370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:54.588171 29370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:54.588356 29370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:15:54.597234 29370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:54.598469 29370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:54.600121 29370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:54.611661 29376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:54.612316 29375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:54.612954 29378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:54.613682 29370 server_base.cc:1034] running on GCE node
I20240502 14:15:54.614198 29370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:54.615136 29370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:54.616333 29370 hybrid_clock.cc:648] HybridClock initialized: now 1714659354616296 us; error 28 us; skew 500 ppm
I20240502 14:15:54.616616 29370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:54.624788 29370 webserver.cc:460] Webserver started at http://127.21.58.68:44249/ using document root <none> and password file <none>
I20240502 14:15:54.625502 29370 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:54.625609 29370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:54.625931 29370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:54.630918 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "139e612d62744c02bcb5653b7aa1dc29"
format_stamp: "Formatted at 2024-05-02 14:15:54 on dist-test-slave-wtgm"
I20240502 14:15:54.631759 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "139e612d62744c02bcb5653b7aa1dc29"
format_stamp: "Formatted at 2024-05-02 14:15:54 on dist-test-slave-wtgm"
I20240502 14:15:54.636734 29370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240502 14:15:54.639978 29384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.641167 29370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 14:15:54.641347 29370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "139e612d62744c02bcb5653b7aa1dc29"
format_stamp: "Formatted at 2024-05-02 14:15:54 on dist-test-slave-wtgm"
I20240502 14:15:54.641530 29370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:54.665352 29370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:54.666132 29370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:54.666389 29370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:54.667318 29370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:54.668720 29370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:54.668812 29370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.668942 29370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:54.669014 29370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.699929 29370 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:46509
I20240502 14:15:54.699976 29497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:46509 every 8 connection(s)
I20240502 14:15:54.701140 29370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:15:54.707780 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 29370
I20240502 14:15:54.707965 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:15:54.711375 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:0
--local_ip_for_outbound_sockets=127.21.58.69
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:54.713630 29498 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:54.713907 29498 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:54.714697 29498 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:54.716051 28915 ts_manager.cc:194] Registered new tserver with Master: 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:15:54.716794 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:59199
I20240502 14:15:54.812398 29235 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
W20240502 14:15:55.043648 29501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:55.043912 29501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:55.044044 29501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:55.051227 29501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:55.051347 29501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:55.051528 29501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:15:55.059652 29501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:55.060881 29501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:55.062554 29501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:55.073756 29506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:55.074319 29507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:55.074919 29509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:55.075562 29501 server_base.cc:1034] running on GCE node
I20240502 14:15:55.076087 29501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:55.076890 29501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:55.078090 29501 hybrid_clock.cc:648] HybridClock initialized: now 1714659355078047 us; error 38 us; skew 500 ppm
I20240502 14:15:55.078368 29501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:55.080454 29501 webserver.cc:460] Webserver started at http://127.21.58.69:46643/ using document root <none> and password file <none>
I20240502 14:15:55.081144 29501 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:55.081250 29501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:55.081569 29501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:55.083971 29501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ede5e1f11f254824a335e1e43f40b6d1"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.084657 29501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ede5e1f11f254824a335e1e43f40b6d1"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.089460 29501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:15:55.092519 29515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.093683 29501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:55.093844 29501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ede5e1f11f254824a335e1e43f40b6d1"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.094038 29501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:55.110824 29501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:55.111651 29501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:55.111908 29501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:55.112810 29501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:55.114279 29501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:55.114372 29501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.114481 29501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:55.114550 29501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.144364 29501 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:36535
I20240502 14:15:55.144407 29627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:36535 every 8 connection(s)
I20240502 14:15:55.145673 29501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:15:55.151374 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 29501
I20240502 14:15:55.151551 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:15:55.154649 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.70:0
--local_ip_for_outbound_sockets=127.21.58.70
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:55.158804 29628 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:55.159164 29628 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:55.159931 29628 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:55.161258 28915 ts_manager.cc:194] Registered new tserver with Master: ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535)
I20240502 14:15:55.162022 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:43311
I20240502 14:15:55.259289 29367 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
W20240502 14:15:55.479871 29631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:55.480149 29631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:55.480232 29631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:55.487349 29631 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:55.487497 29631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:55.487617 29631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.70
I20240502 14:15:55.496299 29631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.70
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:55.497537 29631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:55.499259 29631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:55.510605 29636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:55.511245 29639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:55.511735 29637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:55.512434 29631 server_base.cc:1034] running on GCE node
I20240502 14:15:55.513000 29631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:55.513859 29631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:55.515113 29631 hybrid_clock.cc:648] HybridClock initialized: now 1714659355515035 us; error 44 us; skew 500 ppm
I20240502 14:15:55.515451 29631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:55.517576 29631 webserver.cc:460] Webserver started at http://127.21.58.70:34405/ using document root <none> and password file <none>
I20240502 14:15:55.518344 29631 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:55.518440 29631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:55.518745 29631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:55.521246 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3bb852e0a21446bbbe824bdefb16036c"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.521955 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3bb852e0a21446bbbe824bdefb16036c"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.526950 29631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240502 14:15:55.530161 29645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.531380 29631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:55.531522 29631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3bb852e0a21446bbbe824bdefb16036c"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.531694 29631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:55.547122 29631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:55.547927 29631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:55.548169 29631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:55.549034 29631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:55.550343 29631 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:55.550423 29631 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.550527 29631 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:55.550596 29631 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.577389 29631 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.70:41819
I20240502 14:15:55.577437 29758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.70:41819 every 8 connection(s)
I20240502 14:15:55.578505 29631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:15:55.583774 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 29631
I20240502 14:15:55.583931 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:15:55.587239 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.71:0
--local_ip_for_outbound_sockets=127.21.58.71
--webserver_interface=127.21.58.71
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:55.591166 29759 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:55.591465 29759 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:55.592309 29759 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:55.593815 28915 ts_manager.cc:194] Registered new tserver with Master: 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819)
I20240502 14:15:55.594803 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.70:39823
I20240502 14:15:55.719170 29498 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
W20240502 14:15:55.917201 29762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:55.917469 29762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:55.917613 29762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:55.924993 29762 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:55.925100 29762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:55.925246 29762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.71
I20240502 14:15:55.933905 29762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.58.71
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.71
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:55.935204 29762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:55.936939 29762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:55.948662 29767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:55.949164 29770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:55.949699 29768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:55.950361 29762 server_base.cc:1034] running on GCE node
I20240502 14:15:55.950868 29762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:55.951782 29762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:55.952966 29762 hybrid_clock.cc:648] HybridClock initialized: now 1714659355952913 us; error 46 us; skew 500 ppm
I20240502 14:15:55.953258 29762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:55.955416 29762 webserver.cc:460] Webserver started at http://127.21.58.71:36361/ using document root <none> and password file <none>
I20240502 14:15:55.956132 29762 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:55.956238 29762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:55.956571 29762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:55.958982 29762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b7dacf388234446b9a11ce750978ca75"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.959697 29762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b7dacf388234446b9a11ce750978ca75"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.964685 29762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240502 14:15:55.967913 29776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.969141 29762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:15:55.969300 29762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b7dacf388234446b9a11ce750978ca75"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:15:55.969501 29762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:55.991309 29762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:55.992149 29762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:55.992410 29762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:55.993352 29762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:55.994772 29762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:55.994866 29762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:55.994978 29762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:55.995052 29762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:56.025290 29762 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.71:44703
I20240502 14:15:56.025336 29890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.71:44703 every 8 connection(s)
I20240502 14:15:56.026559 29762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:15:56.028661 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 29762
I20240502 14:15:56.028829 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:15:56.031798 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.72:0
--local_ip_for_outbound_sockets=127.21.58.72
--webserver_interface=127.21.58.72
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:56.039273 29891 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:56.039593 29891 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:56.040549 29891 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:56.042047 28915 ts_manager.cc:194] Registered new tserver with Master: b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703)
I20240502 14:15:56.042809 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.71:60071
I20240502 14:15:56.164333 29628 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
W20240502 14:15:56.369565 29894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:56.369871 29894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:56.370007 29894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:56.377364 29894 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:56.377503 29894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:56.377653 29894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.72
I20240502 14:15:56.386324 29894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.58.72
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.72
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:56.387662 29894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:56.389421 29894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:56.401389 29899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:56.402055 29902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:56.402729 29900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:56.403439 29894 server_base.cc:1034] running on GCE node
I20240502 14:15:56.404000 29894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:56.404780 29894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:56.405979 29894 hybrid_clock.cc:648] HybridClock initialized: now 1714659356405922 us; error 50 us; skew 500 ppm
I20240502 14:15:56.406284 29894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:56.408362 29894 webserver.cc:460] Webserver started at http://127.21.58.72:46021/ using document root <none> and password file <none>
I20240502 14:15:56.409035 29894 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:56.409135 29894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:56.409435 29894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:56.412048 29894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d653f68cb53b4ed79d5e688987ed8d4c"
format_stamp: "Formatted at 2024-05-02 14:15:56 on dist-test-slave-wtgm"
I20240502 14:15:56.412770 29894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d653f68cb53b4ed79d5e688987ed8d4c"
format_stamp: "Formatted at 2024-05-02 14:15:56 on dist-test-slave-wtgm"
I20240502 14:15:56.417726 29894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:15:56.420832 29908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:56.422010 29894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:56.422150 29894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d653f68cb53b4ed79d5e688987ed8d4c"
format_stamp: "Formatted at 2024-05-02 14:15:56 on dist-test-slave-wtgm"
I20240502 14:15:56.422322 29894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:56.438580 29894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:56.439369 29894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:56.439622 29894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:56.440482 29894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:56.441829 29894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:56.441913 29894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:56.442014 29894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:56.442081 29894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:56.473342 29894 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.72:36549
I20240502 14:15:56.473397 30020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.72:36549 every 8 connection(s)
I20240502 14:15:56.474601 29894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:15:56.484946 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 29894
I20240502 14:15:56.485123 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:15:56.486335 30021 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:15:56.486660 30021 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:56.487622 30021 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:15:56.489027 28915 ts_manager.cc:194] Registered new tserver with Master: d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549)
I20240502 14:15:56.489827 28915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.72:43051
I20240502 14:15:56.500912 21737 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:15:56.517048 21737 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:15:56.517166 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 29501
I20240502 14:15:56.524837 29623 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:56.597434 29759 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
I20240502 14:15:56.605389 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 29501
I20240502 14:15:56.623030 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 29631
I20240502 14:15:56.630920 29754 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:56.710863 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 29631
I20240502 14:15:56.728673 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 29762
I20240502 14:15:56.736771 29885 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:56.815642 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 29762
I20240502 14:15:56.832677 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 29894
I20240502 14:15:56.840700 30016 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:56.922439 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 29894
I20240502 14:16:00.691265 28915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:00.747821 29301 tablet_service.cc:1467] Processing CreateTablet for tablet e6a0a881fcec43b291148f071e4474fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.748989 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a0a881fcec43b291148f071e4474fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.748762 29432 tablet_service.cc:1467] Processing CreateTablet for tablet e6a0a881fcec43b291148f071e4474fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.750092 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a0a881fcec43b291148f071e4474fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.746564 29300 tablet_service.cc:1467] Processing CreateTablet for tablet fd4e4241d42d483ab6203606a4d92141 (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.751901 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4e4241d42d483ab6203606a4d92141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.752024 29431 tablet_service.cc:1467] Processing CreateTablet for tablet 3fde0741532a4067a4806f4bc0b0d465 (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.752503 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fde0741532a4067a4806f4bc0b0d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.755945 29169 tablet_service.cc:1467] Processing CreateTablet for tablet e6a0a881fcec43b291148f071e4474fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.756570 29168 tablet_service.cc:1467] Processing CreateTablet for tablet fd4e4241d42d483ab6203606a4d92141 (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.756937 29167 tablet_service.cc:1467] Processing CreateTablet for tablet 3fde0741532a4067a4806f4bc0b0d465 (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.757424 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4e4241d42d483ab6203606a4d92141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.757839 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fde0741532a4067a4806f4bc0b0d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.759311 29169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a0a881fcec43b291148f071e4474fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.767769 29036 tablet_service.cc:1467] Processing CreateTablet for tablet 3fde0741532a4067a4806f4bc0b0d465 (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.768949 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fde0741532a4067a4806f4bc0b0d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.772636 30050 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:00.766553 29037 tablet_service.cc:1467] Processing CreateTablet for tablet fd4e4241d42d483ab6203606a4d92141 (DEFAULT_TABLE table=rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:00.774487 29037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4e4241d42d483ab6203606a4d92141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:00.777151 30050 tablet_bootstrap.cc:654] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.778528 30050 log.cc:824] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:00.799896 30051 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:00.801259 30053 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:00.803337 30054 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:00.804190 30051 tablet_bootstrap.cc:654] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.804983 30053 tablet_bootstrap.cc:654] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.805495 30051 log.cc:824] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:00.806295 30053 log.cc:824] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:00.807046 30054 tablet_bootstrap.cc:654] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.811491 30050 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:00.811930 30050 ts_tablet_manager.cc:1367] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.015s
I20240502 14:16:00.815901 30054 log.cc:824] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:00.816449 30053 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:00.816783 30053 ts_tablet_manager.cc:1367] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20240502 14:16:00.819159 30051 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:00.820056 30051 ts_tablet_manager.cc:1367] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.013s
I20240502 14:16:00.821358 30050 raft_consensus.cc:348] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.821923 30050 raft_consensus.cc:374] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.822070 30050 raft_consensus.cc:729] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.823005 30050 consensus_queue.cc:260] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.828267 30053 raft_consensus.cc:348] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:00.828684 30050 ts_tablet_manager.cc:1398] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
I20240502 14:16:00.828707 30053 raft_consensus.cc:374] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.828863 30053 raft_consensus.cc:729] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.829196 30050 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:00.829519 30053 consensus_queue.cc:260] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:00.830914 30050 tablet_bootstrap.cc:654] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.831385 30054 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:00.831683 30054 ts_tablet_manager.cc:1367] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.006s
I20240502 14:16:00.836218 30050 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:00.836388 30050 ts_tablet_manager.cc:1367] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240502 14:16:00.837270 30053 ts_tablet_manager.cc:1398] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20240502 14:16:00.837405 30050 raft_consensus.cc:348] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:00.837621 30050 raft_consensus.cc:374] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.837702 30053 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:00.837754 30050 raft_consensus.cc:729] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.838033 30050 consensus_queue.cc:260] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:00.839422 30053 tablet_bootstrap.cc:654] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.841027 30050 ts_tablet_manager.cc:1398] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:16:00.842505 30051 raft_consensus.cc:348] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.843160 30051 raft_consensus.cc:374] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.843309 30051 raft_consensus.cc:729] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.844036 30051 consensus_queue.cc:260] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.845255 30054 raft_consensus.cc:348] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.845796 30054 raft_consensus.cc:374] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.845916 30054 raft_consensus.cc:729] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.846716 30054 consensus_queue.cc:260] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.848610 30051 ts_tablet_manager.cc:1398] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.028s user 0.014s sys 0.002s
I20240502 14:16:00.849043 30051 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:00.849285 30054 ts_tablet_manager.cc:1398] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.017s user 0.010s sys 0.003s
I20240502 14:16:00.849689 30054 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:00.850733 30053 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:00.850795 30051 tablet_bootstrap.cc:654] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.850878 30053 ts_tablet_manager.cc:1367] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 14:16:00.851363 30054 tablet_bootstrap.cc:654] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.852020 30053 raft_consensus.cc:348] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.852288 30053 raft_consensus.cc:374] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.852389 30053 raft_consensus.cc:729] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.852653 30053 consensus_queue.cc:260] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.853228 30053 ts_tablet_manager.cc:1398] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:00.855518 30054 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:00.855665 30054 ts_tablet_manager.cc:1367] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240502 14:16:00.856006 30051 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:00.856599 30054 raft_consensus.cc:348] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.856839 30054 raft_consensus.cc:374] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.856935 30054 raft_consensus.cc:729] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.857056 30051 ts_tablet_manager.cc:1367] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 14:16:00.857156 30054 consensus_queue.cc:260] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.857686 30054 ts_tablet_manager.cc:1398] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:00.857967 30051 raft_consensus.cc:348] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:00.858162 30051 raft_consensus.cc:374] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.858258 30051 raft_consensus.cc:729] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.858446 30051 consensus_queue.cc:260] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:00.858881 30051 ts_tablet_manager.cc:1398] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:00.859144 30051 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:00.860379 30051 tablet_bootstrap.cc:654] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:00.862133 30051 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:00.862270 30051 ts_tablet_manager.cc:1367] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:00.862835 30051 raft_consensus.cc:348] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.863005 30051 raft_consensus.cc:374] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:00.863072 30051 raft_consensus.cc:729] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:00.863317 30051 consensus_queue.cc:260] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.863744 30051 ts_tablet_manager.cc:1398] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:16:00.893352 30062 raft_consensus.cc:482] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:00.893664 30062 raft_consensus.cc:504] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.895465 30062 leader_election.cc:288] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:00.897044 30063 raft_consensus.cc:482] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:00.897292 30063 raft_consensus.cc:504] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.898998 30063 leader_election.cc:288] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:00.909720 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4e4241d42d483ab6203606a4d92141" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:00.910391 29321 raft_consensus.cc:2454] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:00.911295 29121 leader_election.cc:302] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:00.912459 30062 raft_consensus.cc:2781] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:00.912616 30062 raft_consensus.cc:482] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:00.912707 30062 raft_consensus.cc:3037] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:00.915890 30062 raft_consensus.cc:504] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.916873 30062 leader_election.cc:288] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:00.916929 29452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fde0741532a4067a4806f4bc0b0d465" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:00.917485 29452 raft_consensus.cc:2454] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:00.918165 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4e4241d42d483ab6203606a4d92141" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:00.918360 29321 raft_consensus.cc:3037] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:00.918507 29056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4e4241d42d483ab6203606a4d92141" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:00.917855 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4e4241d42d483ab6203606a4d92141" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:00.918926 28992 leader_election.cc:302] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:00.919128 29057 raft_consensus.cc:2454] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:00.919521 30063 raft_consensus.cc:2781] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:00.919657 30063 raft_consensus.cc:482] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:00.919762 30063 raft_consensus.cc:3037] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:00.921473 29321 raft_consensus.cc:2454] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:00.922335 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fde0741532a4067a4806f4bc0b0d465" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:00.922621 30063 raft_consensus.cc:504] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.922729 29189 raft_consensus.cc:2454] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:00.923278 30063 leader_election.cc:288] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:00.923538 29121 leader_election.cc:302] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters: 9329e8a7a2d1453d90b76b5965315837
I20240502 14:16:00.923735 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fde0741532a4067a4806f4bc0b0d465" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:00.923847 30062 raft_consensus.cc:2781] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:00.923930 29189 raft_consensus.cc:3037] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:00.924367 30062 raft_consensus.cc:686] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Becoming Leader. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Running, Role: LEADER
I20240502 14:16:00.924633 29452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fde0741532a4067a4806f4bc0b0d465" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:00.924839 29452 raft_consensus.cc:3037] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:00.925383 30062 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.926976 29189 raft_consensus.cc:2454] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:00.927093 29452 raft_consensus.cc:2454] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:00.927415 28993 leader_election.cc:302] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:00.927747 30063 raft_consensus.cc:2781] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:00.930248 28914 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbabfd51061b4e509d907a3622febd2e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:00.935323 30063 raft_consensus.cc:686] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:00.935777 30063 consensus_queue.cc:237] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:00.939741 28914 catalog_manager.cc:5484] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:00.957624 29499 tablet.cc:2369] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:16:00.995194 29368 tablet.cc:2369] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:16:01.021102 30058 raft_consensus.cc:482] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:01.021299 30058 raft_consensus.cc:504] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:01.022862 30058 leader_election.cc:288] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:01.031245 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a0a881fcec43b291148f071e4474fc" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:01.031329 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a0a881fcec43b291148f071e4474fc" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:01.031482 29189 raft_consensus.cc:2454] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:01.031576 29321 raft_consensus.cc:2454] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:01.032028 29388 leader_election.cc:302] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:01.032392 30058 raft_consensus.cc:2781] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:01.032500 30058 raft_consensus.cc:482] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:01.032588 30058 raft_consensus.cc:3037] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:01.034559 30058 raft_consensus.cc:504] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:01.035218 30058 leader_election.cc:288] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:01.035558 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a0a881fcec43b291148f071e4474fc" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:01.035691 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a0a881fcec43b291148f071e4474fc" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:01.035811 29189 raft_consensus.cc:3037] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:01.035917 29321 raft_consensus.cc:3037] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:01.038599 29321 raft_consensus.cc:2454] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:01.038599 29189 raft_consensus.cc:2454] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:01.038998 29385 leader_election.cc:302] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:01.039307 30058 raft_consensus.cc:2781] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:01.039669 30058 raft_consensus.cc:686] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:01.040148 30058 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:01.044186 28914 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:01.048674 29236 tablet.cc:2369] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:16:01.093403 29103 tablet.cc:2369] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:16:01.312161 29057 raft_consensus.cc:3037] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:01.316449 30062 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:01.324339 30062 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:01.396075 30079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:01.396373 30079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:01.403960 30079 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:01.426538 30058 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:01.434401 30099 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 14:16:01.472441 30063 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:01.478575 30114 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 231.246 ms
time per row: 0.115623 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:16:02.691445 28912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:02.699832 29431 tablet_service.cc:1467] Processing CreateTablet for tablet 71f047f74165489bacb4f72ab981b447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.700310 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f047f74165489bacb4f72ab981b447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.700212 29036 tablet_service.cc:1467] Processing CreateTablet for tablet 71f047f74165489bacb4f72ab981b447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.700433 29037 tablet_service.cc:1467] Processing CreateTablet for tablet f4dbd9bc62484dd1998080d289db098f (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.700639 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f047f74165489bacb4f72ab981b447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.701448 29037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dbd9bc62484dd1998080d289db098f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.701515 29432 tablet_service.cc:1467] Processing CreateTablet for tablet f4dbd9bc62484dd1998080d289db098f (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.701867 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dbd9bc62484dd1998080d289db098f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.704285 29301 tablet_service.cc:1467] Processing CreateTablet for tablet 71f047f74165489bacb4f72ab981b447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.704721 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f047f74165489bacb4f72ab981b447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.704802 29300 tablet_service.cc:1467] Processing CreateTablet for tablet f4dbd9bc62484dd1998080d289db098f (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.705147 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dbd9bc62484dd1998080d289db098f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.711167 29299 tablet_service.cc:1467] Processing CreateTablet for tablet 1f17889f58a34bc0afa40a246e5debc6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.711501 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f17889f58a34bc0afa40a246e5debc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.711483 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 1f17889f58a34bc0afa40a246e5debc6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.711858 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f17889f58a34bc0afa40a246e5debc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.716188 30150 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:02.716253 29168 tablet_service.cc:1467] Processing CreateTablet for tablet 1f17889f58a34bc0afa40a246e5debc6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:02.716627 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f17889f58a34bc0afa40a246e5debc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:02.718909 30150 tablet_bootstrap.cc:654] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.721022 30152 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:02.723099 30152 tablet_bootstrap.cc:654] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.724035 30151 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:02.725512 30150 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:02.725709 30150 ts_tablet_manager.cc:1367] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:16:02.726426 30151 tablet_bootstrap.cc:654] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.726835 30150 raft_consensus.cc:348] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.727141 30150 raft_consensus.cc:374] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.727269 30150 raft_consensus.cc:729] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.728052 30150 consensus_queue.cc:260] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.730044 30150 ts_tablet_manager.cc:1398] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:16:02.735018 30151 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:02.735235 30151 ts_tablet_manager.cc:1367] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240502 14:16:02.736040 30156 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:02.736641 30151 raft_consensus.cc:348] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.736936 30151 raft_consensus.cc:374] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.737049 30151 raft_consensus.cc:729] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.737368 30151 consensus_queue.cc:260] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.737746 30156 tablet_bootstrap.cc:654] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.739775 30150 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:02.741768 30150 tablet_bootstrap.cc:654] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.748461 30151 ts_tablet_manager.cc:1398] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.013s user 0.000s sys 0.012s
I20240502 14:16:02.748668 30150 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:02.748776 30151 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:02.748819 30150 ts_tablet_manager.cc:1367] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240502 14:16:02.749446 30150 raft_consensus.cc:348] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.749610 30150 raft_consensus.cc:374] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.749688 30150 raft_consensus.cc:729] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.749912 30150 consensus_queue.cc:260] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.750377 30150 ts_tablet_manager.cc:1398] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:16:02.750563 30152 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:02.750654 30150 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:02.750701 30152 ts_tablet_manager.cc:1367] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.023s
I20240502 14:16:02.751502 30152 raft_consensus.cc:348] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.751725 30152 raft_consensus.cc:374] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.751834 30152 raft_consensus.cc:729] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.752079 30152 consensus_queue.cc:260] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.752277 30150 tablet_bootstrap.cc:654] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.752596 30152 ts_tablet_manager.cc:1398] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:02.752872 30152 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:02.753024 30156 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:02.753175 30156 ts_tablet_manager.cc:1367] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240502 14:16:02.753852 30156 raft_consensus.cc:348] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.754009 30156 raft_consensus.cc:374] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.754106 30156 raft_consensus.cc:729] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.754328 30156 consensus_queue.cc:260] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.754460 30152 tablet_bootstrap.cc:654] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.754863 30156 ts_tablet_manager.cc:1398] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:02.755998 30150 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:02.756119 30150 ts_tablet_manager.cc:1367] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:16:02.756402 30152 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:02.756556 30152 ts_tablet_manager.cc:1367] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:16:02.756685 30150 raft_consensus.cc:348] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.756840 30150 raft_consensus.cc:374] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.756927 30150 raft_consensus.cc:729] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.757129 30150 consensus_queue.cc:260] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.757253 30152 raft_consensus.cc:348] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.757463 30152 raft_consensus.cc:374] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.757545 30152 raft_consensus.cc:729] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.757604 30150 ts_tablet_manager.cc:1398] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:16:02.757762 30152 consensus_queue.cc:260] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.758318 30152 ts_tablet_manager.cc:1398] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:02.759120 30151 tablet_bootstrap.cc:654] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.766829 30151 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:02.766948 30151 ts_tablet_manager.cc:1367] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20240502 14:16:02.767498 30151 raft_consensus.cc:348] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.767657 30151 raft_consensus.cc:374] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.767720 30151 raft_consensus.cc:729] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.767906 30151 consensus_queue.cc:260] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.768260 30151 ts_tablet_manager.cc:1398] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:16:02.768541 30151 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:02.769665 30151 tablet_bootstrap.cc:654] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:02.771222 30151 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:02.771345 30151 ts_tablet_manager.cc:1367] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:02.771821 30151 raft_consensus.cc:348] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.771953 30151 raft_consensus.cc:374] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:02.772056 30151 raft_consensus.cc:729] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:02.772248 30151 consensus_queue.cc:260] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.772725 30151 ts_tablet_manager.cc:1398] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:16:02.843932 30098 raft_consensus.cc:482] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:02.844101 30098 raft_consensus.cc:504] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.844849 30098 leader_election.cc:288] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:02.845707 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dbd9bc62484dd1998080d289db098f" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:02.845924 29321 raft_consensus.cc:2454] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:02.846364 29385 leader_election.cc:302] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters:
I20240502 14:16:02.846642 30098 raft_consensus.cc:2781] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:02.846748 30098 raft_consensus.cc:482] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:02.846837 30098 raft_consensus.cc:3037] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.849375 30098 raft_consensus.cc:504] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.850070 30098 leader_election.cc:288] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:02.850347 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dbd9bc62484dd1998080d289db098f" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:02.850490 29321 raft_consensus.cc:3037] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.851527 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dbd9bc62484dd1998080d289db098f" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:02.851630 29056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dbd9bc62484dd1998080d289db098f" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:02.851794 29057 raft_consensus.cc:2454] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:02.852639 29321 raft_consensus.cc:2454] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:02.853022 29385 leader_election.cc:302] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters: 9329e8a7a2d1453d90b76b5965315837
I20240502 14:16:02.853286 30098 raft_consensus.cc:2781] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:02.853623 30098 raft_consensus.cc:686] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:02.853924 30098 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:02.856447 28914 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:02.862007 30098 raft_consensus.cc:482] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:02.862185 30098 raft_consensus.cc:504] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.862655 30098 leader_election.cc:288] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:02.863142 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f047f74165489bacb4f72ab981b447" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:02.863144 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f047f74165489bacb4f72ab981b447" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:02.863363 29321 raft_consensus.cc:2454] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:02.863364 29057 raft_consensus.cc:2454] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:02.863715 29387 leader_election.cc:302] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:02.863976 30098 raft_consensus.cc:2781] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:02.864050 30098 raft_consensus.cc:482] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:02.864133 30098 raft_consensus.cc:3037] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.865870 30098 raft_consensus.cc:504] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.866312 30098 leader_election.cc:288] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:02.866631 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f047f74165489bacb4f72ab981b447" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:02.866703 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f047f74165489bacb4f72ab981b447" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:02.866818 29057 raft_consensus.cc:3037] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.866887 29321 raft_consensus.cc:3037] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.868698 29057 raft_consensus.cc:2454] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:02.868710 29321 raft_consensus.cc:2454] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:02.869014 29387 leader_election.cc:302] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:02.869277 30098 raft_consensus.cc:2781] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:02.869446 30098 raft_consensus.cc:686] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:02.869676 30098 consensus_queue.cc:237] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:02.871793 28914 catalog_manager.cc:5484] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:02.969295 30157 raft_consensus.cc:482] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:02.969481 30157 raft_consensus.cc:504] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.971024 30157 leader_election.cc:288] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:02.977384 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f17889f58a34bc0afa40a246e5debc6" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:02.977627 29189 raft_consensus.cc:2454] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:02.977766 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f17889f58a34bc0afa40a246e5debc6" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:02.978018 29057 raft_consensus.cc:2454] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:02.978077 29257 leader_election.cc:302] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:02.978475 30157 raft_consensus.cc:2781] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:02.978614 30157 raft_consensus.cc:482] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:02.978701 30157 raft_consensus.cc:3037] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.980639 30157 raft_consensus.cc:504] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.981211 30157 leader_election.cc:288] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:02.981676 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f17889f58a34bc0afa40a246e5debc6" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:02.981681 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f17889f58a34bc0afa40a246e5debc6" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:02.981885 29057 raft_consensus.cc:3037] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.981915 29189 raft_consensus.cc:3037] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:02.983719 29057 raft_consensus.cc:2454] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:02.983770 29189 raft_consensus.cc:2454] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:02.984118 29256 leader_election.cc:302] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:02.984408 30157 raft_consensus.cc:2781] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:02.984789 30157 raft_consensus.cc:686] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Running, Role: LEADER
I20240502 14:16:02.985260 30157 consensus_queue.cc:237] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:02.988584 28912 catalog_manager.cc:5484] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:03.316390 30098 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:03.320921 29057 raft_consensus.cc:3037] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:03.323731 30098 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:03.363009 30174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:03.363402 30174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:03.370443 30174 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:16:03.425621 30098 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:03.430657 30157 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:03.442749 30098 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:03.483486 30157 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 300.959 ms
time per row: 0.150479 ms
I20240502 14:16:03.811213 28914 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:03.819183 29168 tablet_service.cc:1467] Processing CreateTablet for tablet e66f9bb135bf4007b1864b63a389fac3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.819630 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f9bb135bf4007b1864b63a389fac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.819698 29431 tablet_service.cc:1467] Processing CreateTablet for tablet e66f9bb135bf4007b1864b63a389fac3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.820204 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f9bb135bf4007b1864b63a389fac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.821094 29432 tablet_service.cc:1467] Processing CreateTablet for tablet 44d94461e22c4ae182afa2a15f3db198 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.821470 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d94461e22c4ae182afa2a15f3db198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.821343 29430 tablet_service.cc:1467] Processing CreateTablet for tablet 981ec6067d624100bddba4383b417c50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.821772 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981ec6067d624100bddba4383b417c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.822405 29300 tablet_service.cc:1467] Processing CreateTablet for tablet 44d94461e22c4ae182afa2a15f3db198 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.822822 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d94461e22c4ae182afa2a15f3db198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.823985 29036 tablet_service.cc:1467] Processing CreateTablet for tablet 981ec6067d624100bddba4383b417c50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.824410 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981ec6067d624100bddba4383b417c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.825438 29035 tablet_service.cc:1467] Processing CreateTablet for tablet e66f9bb135bf4007b1864b63a389fac3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.825873 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f9bb135bf4007b1864b63a389fac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.828002 29167 tablet_service.cc:1467] Processing CreateTablet for tablet 44d94461e22c4ae182afa2a15f3db198 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.828354 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d94461e22c4ae182afa2a15f3db198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.831337 29169 tablet_service.cc:1467] Processing CreateTablet for tablet 981ec6067d624100bddba4383b417c50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:03.831674 29169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981ec6067d624100bddba4383b417c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.832114 30236 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:03.834167 30236 tablet_bootstrap.cc:654] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.836328 30236 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:03.836467 30236 ts_tablet_manager.cc:1367] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 14:16:03.837215 30236 raft_consensus.cc:348] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.837441 30236 raft_consensus.cc:374] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.837532 30236 raft_consensus.cc:729] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.837764 30236 consensus_queue.cc:260] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.838318 30236 ts_tablet_manager.cc:1398] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:03.845408 30236 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:03.847648 30236 tablet_bootstrap.cc:654] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.852007 30240 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:03.854374 30239 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:03.855463 30240 tablet_bootstrap.cc:654] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.856109 30239 tablet_bootstrap.cc:654] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.856403 30236 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:03.856658 30236 ts_tablet_manager.cc:1367] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240502 14:16:03.857756 30236 raft_consensus.cc:348] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.858121 30236 raft_consensus.cc:374] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.858242 30236 raft_consensus.cc:729] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.858618 30237 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:03.858570 30236 consensus_queue.cc:260] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.859468 30236 ts_tablet_manager.cc:1398] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:16:03.859997 30239 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:03.860128 30239 ts_tablet_manager.cc:1367] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 14:16:03.860771 30237 tablet_bootstrap.cc:654] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.860843 30239 raft_consensus.cc:348] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.861048 30239 raft_consensus.cc:374] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.861147 30239 raft_consensus.cc:729] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.861361 30239 consensus_queue.cc:260] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.861832 30239 ts_tablet_manager.cc:1398] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:03.862138 30239 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:03.863430 30239 tablet_bootstrap.cc:654] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.865108 30239 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:03.865257 30239 ts_tablet_manager.cc:1367] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:03.866626 30240 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:03.866081 30239 raft_consensus.cc:348] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.866789 30240 ts_tablet_manager.cc:1367] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240502 14:16:03.866806 30239 raft_consensus.cc:374] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.866873 30239 raft_consensus.cc:729] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.867120 30239 consensus_queue.cc:260] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.867619 30239 ts_tablet_manager.cc:1398] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:16:03.867640 30240 raft_consensus.cc:348] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.867821 30240 raft_consensus.cc:374] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.867893 30239 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:03.867908 30240 raft_consensus.cc:729] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.868149 30240 consensus_queue.cc:260] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.869271 30240 ts_tablet_manager.cc:1398] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:03.869848 30239 tablet_bootstrap.cc:654] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.870579 30237 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:03.870759 30237 ts_tablet_manager.cc:1367] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240502 14:16:03.871678 30237 raft_consensus.cc:348] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.871958 30237 raft_consensus.cc:374] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.872109 30237 raft_consensus.cc:729] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.872421 30237 consensus_queue.cc:260] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.872926 30239 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:03.873054 30239 ts_tablet_manager.cc:1367] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 14:16:03.873591 30237 ts_tablet_manager.cc:1398] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:03.873667 30239 raft_consensus.cc:348] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.873860 30239 raft_consensus.cc:374] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.873901 30237 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:03.873963 30239 raft_consensus.cc:729] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.874177 30239 consensus_queue.cc:260] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.875001 30239 ts_tablet_manager.cc:1398] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:03.876082 30237 tablet_bootstrap.cc:654] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.877966 30237 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:03.878257 30237 ts_tablet_manager.cc:1367] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:16:03.878975 30237 raft_consensus.cc:348] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.879209 30237 raft_consensus.cc:374] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.879287 30237 raft_consensus.cc:729] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.879503 30237 consensus_queue.cc:260] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.879911 30237 ts_tablet_manager.cc:1398] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:03.880239 30237 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:03.881436 30237 tablet_bootstrap.cc:654] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:03.883186 30237 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:03.883322 30237 ts_tablet_manager.cc:1367] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:03.883864 30237 raft_consensus.cc:348] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.884020 30237 raft_consensus.cc:374] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:03.884112 30237 raft_consensus.cc:729] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.884295 30237 consensus_queue.cc:260] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.884693 30237 ts_tablet_manager.cc:1398] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:16:03.899209 30071 raft_consensus.cc:482] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:03.899390 30071 raft_consensus.cc:504] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.900162 30071 leader_election.cc:288] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:03.900714 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f9bb135bf4007b1864b63a389fac3" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:03.900987 29057 raft_consensus.cc:2454] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:03.901480 29124 leader_election.cc:302] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:03.901808 30071 raft_consensus.cc:2781] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:03.901993 30071 raft_consensus.cc:482] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:03.902129 30071 raft_consensus.cc:3037] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:03.904814 30071 raft_consensus.cc:504] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.905390 30071 leader_election.cc:288] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Requested vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:03.905771 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f9bb135bf4007b1864b63a389fac3" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:03.905953 29057 raft_consensus.cc:3037] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:03.907413 29452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f9bb135bf4007b1864b63a389fac3" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:03.907505 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f9bb135bf4007b1864b63a389fac3" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:03.907680 29452 raft_consensus.cc:2454] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:03.908618 29057 raft_consensus.cc:2454] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:03.908934 29124 leader_election.cc:302] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters: 139e612d62744c02bcb5653b7aa1dc29
I20240502 14:16:03.909200 30071 raft_consensus.cc:2781] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:03.909541 30071 raft_consensus.cc:686] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Becoming Leader. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Running, Role: LEADER
I20240502 14:16:03.909885 30071 consensus_queue.cc:237] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:03.912312 28914 catalog_manager.cc:5484] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbabfd51061b4e509d907a3622febd2e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:03.948622 30071 raft_consensus.cc:482] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:03.948843 30071 raft_consensus.cc:504] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.949395 30071 leader_election.cc:288] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:03.949837 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:03.949858 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:03.950052 29451 raft_consensus.cc:2454] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:03.950069 29321 raft_consensus.cc:2454] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:03.950452 29124 leader_election.cc:302] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:03.950722 30071 raft_consensus.cc:2781] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:03.950820 30071 raft_consensus.cc:482] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:03.950925 30071 raft_consensus.cc:3037] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:03.952877 30071 raft_consensus.cc:504] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.953431 30071 leader_election.cc:288] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:03.953760 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:03.953819 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:03.953935 29321 raft_consensus.cc:3037] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:03.954002 29451 raft_consensus.cc:3037] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:03.955827 29321 raft_consensus.cc:2454] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:03.955883 29451 raft_consensus.cc:2454] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:03.956216 29121 leader_election.cc:302] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:03.956470 30071 raft_consensus.cc:2781] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:03.956615 30071 raft_consensus.cc:686] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Becoming Leader. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Running, Role: LEADER
I20240502 14:16:03.956878 30071 consensus_queue.cc:237] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:03.959308 28914 catalog_manager.cc:5484] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbabfd51061b4e509d907a3622febd2e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:04.048216 30071 raft_consensus.cc:482] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:04.048403 30071 raft_consensus.cc:504] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:04.049063 30071 leader_election.cc:288] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:04.049607 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981ec6067d624100bddba4383b417c50" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:04.049655 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981ec6067d624100bddba4383b417c50" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:04.049858 29451 raft_consensus.cc:2454] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:04.049892 29057 raft_consensus.cc:2454] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:04.050246 29124 leader_election.cc:302] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:04.050535 30071 raft_consensus.cc:2781] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:04.050642 30071 raft_consensus.cc:482] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:04.050731 30071 raft_consensus.cc:3037] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:04.052563 30071 raft_consensus.cc:504] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:04.053035 30071 leader_election.cc:288] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Requested vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:04.053459 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981ec6067d624100bddba4383b417c50" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:04.053543 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981ec6067d624100bddba4383b417c50" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:04.053642 29057 raft_consensus.cc:3037] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:04.053728 29451 raft_consensus.cc:3037] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:04.055513 29451 raft_consensus.cc:2454] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:04.055773 29057 raft_consensus.cc:2454] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:04.055852 29124 leader_election.cc:302] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:04.056159 30071 raft_consensus.cc:2781] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:04.056283 30071 raft_consensus.cc:686] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Becoming Leader. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Running, Role: LEADER
I20240502 14:16:04.056533 30071 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:04.059199 28914 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbabfd51061b4e509d907a3622febd2e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:04.311362 29451 raft_consensus.cc:3037] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:04.314036 30071 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:04.319675 30251 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:04.359478 30251 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:04.365283 30251 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:04.421748 30255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:04.422037 30255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:04.429378 30255 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:16:04.489501 30071 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:04.501837 30251 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.11 ms
time per row: 0.117555 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:16:05.729341 28914 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:05.738310 29300 tablet_service.cc:1467] Processing CreateTablet for tablet 239ee601d297412494cf2b447ef75e9a (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.738492 29168 tablet_service.cc:1467] Processing CreateTablet for tablet 239ee601d297412494cf2b447ef75e9a (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.738776 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee601d297412494cf2b447ef75e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.738919 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee601d297412494cf2b447ef75e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.739832 29432 tablet_service.cc:1467] Processing CreateTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.739832 29430 tablet_service.cc:1467] Processing CreateTablet for tablet a00d9c46bfc64a4085715e4f764b8c96 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.740274 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fcde0e7cbe41f1a521e669b798aa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.740491 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00d9c46bfc64a4085715e4f764b8c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.740881 29301 tablet_service.cc:1467] Processing CreateTablet for tablet a00d9c46bfc64a4085715e4f764b8c96 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.741205 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00d9c46bfc64a4085715e4f764b8c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.743928 29169 tablet_service.cc:1467] Processing CreateTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.744309 29169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fcde0e7cbe41f1a521e669b798aa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.747530 29299 tablet_service.cc:1467] Processing CreateTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.748021 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fcde0e7cbe41f1a521e669b798aa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.751577 29036 tablet_service.cc:1467] Processing CreateTablet for tablet 239ee601d297412494cf2b447ef75e9a (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.752079 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee601d297412494cf2b447ef75e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.755537 29035 tablet_service.cc:1467] Processing CreateTablet for tablet a00d9c46bfc64a4085715e4f764b8c96 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:05.755679 30319 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:05.756011 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00d9c46bfc64a4085715e4f764b8c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:05.757217 30320 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:05.757316 30319 tablet_bootstrap.cc:654] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.759120 30320 tablet_bootstrap.cc:654] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.762181 30318 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:05.764516 30318 tablet_bootstrap.cc:654] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.764546 30320 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:05.764684 30320 ts_tablet_manager.cc:1367] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 14:16:05.765460 30320 raft_consensus.cc:348] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.765691 30320 raft_consensus.cc:374] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.765786 30320 raft_consensus.cc:729] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.766017 30320 consensus_queue.cc:260] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.767161 30320 ts_tablet_manager.cc:1398] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:05.768636 30318 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:05.768891 30318 ts_tablet_manager.cc:1367] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 14:16:05.769810 30318 raft_consensus.cc:348] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.770045 30318 raft_consensus.cc:374] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.770205 30318 raft_consensus.cc:729] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.770454 30318 consensus_queue.cc:260] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.771274 30318 ts_tablet_manager.cc:1398] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:16:05.773963 30318 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:05.775152 30320 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:05.776258 30320 tablet_bootstrap.cc:654] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.776389 30318 tablet_bootstrap.cc:654] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.777880 30320 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:05.778053 30320 ts_tablet_manager.cc:1367] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:05.778584 30320 raft_consensus.cc:348] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.778760 30320 raft_consensus.cc:374] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.778841 30320 raft_consensus.cc:729] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.779103 30320 consensus_queue.cc:260] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.779767 30320 ts_tablet_manager.cc:1398] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:05.778704 30318 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:05.780283 30318 ts_tablet_manager.cc:1367] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 14:16:05.781266 30318 raft_consensus.cc:348] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.781592 30318 raft_consensus.cc:374] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.781723 30318 raft_consensus.cc:729] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.782125 30318 consensus_queue.cc:260] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.784838 30318 ts_tablet_manager.cc:1398] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:16:05.784900 30324 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:05.787238 30324 tablet_bootstrap.cc:654] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.789286 30319 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:05.789423 30319 ts_tablet_manager.cc:1367] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.017s
I20240502 14:16:05.789448 30324 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:05.789573 30324 ts_tablet_manager.cc:1367] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 14:16:05.790062 30319 raft_consensus.cc:348] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.790248 30319 raft_consensus.cc:374] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.790357 30319 raft_consensus.cc:729] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.790246 30324 raft_consensus.cc:348] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.790448 30324 raft_consensus.cc:374] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.790527 30324 raft_consensus.cc:729] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.790596 30319 consensus_queue.cc:260] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.790732 30324 consensus_queue.cc:260] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.791221 30324 ts_tablet_manager.cc:1398] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:05.791523 30324 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:05.792915 30324 tablet_bootstrap.cc:654] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.795840 30324 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:05.796006 30324 ts_tablet_manager.cc:1367] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:16:05.796603 30324 raft_consensus.cc:348] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.796797 30324 raft_consensus.cc:374] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.796859 30324 raft_consensus.cc:729] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.797067 30324 consensus_queue.cc:260] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.797500 30324 ts_tablet_manager.cc:1398] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:16:05.799311 30319 ts_tablet_manager.cc:1398] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 14:16:05.799608 30319 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:05.801123 30319 tablet_bootstrap.cc:654] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.807958 30319 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:05.808135 30319 ts_tablet_manager.cc:1367] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20240502 14:16:05.808852 30319 raft_consensus.cc:348] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.809003 30319 raft_consensus.cc:374] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.809051 30319 raft_consensus.cc:729] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.809211 30319 consensus_queue.cc:260] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.809593 30319 ts_tablet_manager.cc:1398] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:16:05.809934 30319 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:05.811313 30319 tablet_bootstrap.cc:654] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:05.812996 30319 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:05.813129 30319 ts_tablet_manager.cc:1367] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:05.813808 30319 raft_consensus.cc:348] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.813989 30319 raft_consensus.cc:374] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:05.814105 30319 raft_consensus.cc:729] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:05.814306 30319 consensus_queue.cc:260] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.814716 30319 ts_tablet_manager.cc:1398] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:16:05.867103 30228 raft_consensus.cc:482] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:05.867276 30228 raft_consensus.cc:504] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.867900 30228 leader_election.cc:288] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:05.868407 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fcde0e7cbe41f1a521e669b798aa99" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:05.868397 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fcde0e7cbe41f1a521e669b798aa99" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:05.868619 29321 raft_consensus.cc:2454] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:05.868639 29189 raft_consensus.cc:2454] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:05.869060 29388 leader_election.cc:302] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:05.869331 30228 raft_consensus.cc:2781] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:05.869446 30228 raft_consensus.cc:482] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:05.869537 30228 raft_consensus.cc:3037] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:05.871552 30228 raft_consensus.cc:504] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.872143 30228 leader_election.cc:288] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:05.872493 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fcde0e7cbe41f1a521e669b798aa99" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:05.872630 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fcde0e7cbe41f1a521e669b798aa99" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:05.872704 29189 raft_consensus.cc:3037] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:05.872849 29321 raft_consensus.cc:3037] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:05.875542 29189 raft_consensus.cc:2454] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:05.875542 29321 raft_consensus.cc:2454] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:05.875918 29385 leader_election.cc:302] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters:
I20240502 14:16:05.876190 30228 raft_consensus.cc:2781] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:05.876329 30228 raft_consensus.cc:686] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:05.876581 30228 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:05.879030 28915 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:05.909284 30063 raft_consensus.cc:482] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:05.909454 30063 raft_consensus.cc:504] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.910143 30063 leader_election.cc:288] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:05.910667 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee601d297412494cf2b447ef75e9a" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:05.910962 29189 raft_consensus.cc:2454] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:05.911370 28993 leader_election.cc:302] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:05.911664 30063 raft_consensus.cc:2781] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:05.911873 30063 raft_consensus.cc:482] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:05.912040 30063 raft_consensus.cc:3037] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:05.914377 30063 raft_consensus.cc:504] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.915144 30063 leader_election.cc:288] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:05.915549 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee601d297412494cf2b447ef75e9a" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:05.915784 29189 raft_consensus.cc:3037] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:05.917335 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee601d297412494cf2b447ef75e9a" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:05.917402 29320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee601d297412494cf2b447ef75e9a" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:05.917598 29321 raft_consensus.cc:2454] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:05.918324 29189 raft_consensus.cc:2454] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:05.918649 28993 leader_election.cc:302] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters: 1b3c20f7bfd548769f6bd40148590b74
I20240502 14:16:05.918898 30063 raft_consensus.cc:2781] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:05.919260 30063 raft_consensus.cc:686] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:05.919557 30063 consensus_queue.cc:237] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.921891 28915 catalog_manager.cc:5484] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:05.993325 30206 raft_consensus.cc:482] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:05.993489 30206 raft_consensus.cc:504] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.994238 30206 leader_election.cc:288] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:05.995512 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00d9c46bfc64a4085715e4f764b8c96" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:05.995729 29057 raft_consensus.cc:2454] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:05.996186 29256 leader_election.cc:302] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:05.996480 30206 raft_consensus.cc:2781] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:05.996613 30206 raft_consensus.cc:482] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:05.996732 30206 raft_consensus.cc:3037] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:05.999209 30206 raft_consensus.cc:504] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:05.999945 30206 leader_election.cc:288] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Requested vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:06.000548 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00d9c46bfc64a4085715e4f764b8c96" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:06.000770 29057 raft_consensus.cc:3037] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.001354 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00d9c46bfc64a4085715e4f764b8c96" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:06.001441 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00d9c46bfc64a4085715e4f764b8c96" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:06.001585 29451 raft_consensus.cc:2454] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:06.003188 29057 raft_consensus.cc:2454] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:06.003536 29256 leader_election.cc:302] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters: 139e612d62744c02bcb5653b7aa1dc29
I20240502 14:16:06.004017 30206 raft_consensus.cc:2781] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:06.004513 30206 raft_consensus.cc:686] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Running, Role: LEADER
I20240502 14:16:06.004828 30206 consensus_queue.cc:237] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:06.007608 28915 catalog_manager.cc:5484] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:06.325088 30063 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:06.329942 29321 raft_consensus.cc:3037] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.333684 30344 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:06.348510 30228 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:06.356498 30228 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:06.392247 30341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:06.392621 30341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:06.399936 30341 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:06.428874 29451 raft_consensus.cc:3037] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.431389 30206 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 14:16:06.436805 30339 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 222.964 ms
time per row: 0.111482 ms
I20240502 14:16:06.771514 28913 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:06.779182 29430 tablet_service.cc:1467] Processing CreateTablet for tablet 302978e1f6034b3693b3a0abacdc494e (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.779307 29169 tablet_service.cc:1467] Processing CreateTablet for tablet 302978e1f6034b3693b3a0abacdc494e (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.779631 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302978e1f6034b3693b3a0abacdc494e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.779731 29169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302978e1f6034b3693b3a0abacdc494e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.779570 29036 tablet_service.cc:1467] Processing CreateTablet for tablet 302978e1f6034b3693b3a0abacdc494e (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.780055 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302978e1f6034b3693b3a0abacdc494e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.781272 29300 tablet_service.cc:1467] Processing CreateTablet for tablet 96fb0fc5919d42d7bedd770945935b42 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.781682 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fb0fc5919d42d7bedd770945935b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.786271 29299 tablet_service.cc:1467] Processing CreateTablet for tablet 078ffca14316466aa3906d2ff0567c70 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.786599 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078ffca14316466aa3906d2ff0567c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.788458 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 96fb0fc5919d42d7bedd770945935b42 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.788816 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fb0fc5919d42d7bedd770945935b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.791625 29432 tablet_service.cc:1467] Processing CreateTablet for tablet 078ffca14316466aa3906d2ff0567c70 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.791965 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078ffca14316466aa3906d2ff0567c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.794898 30408 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:06.796034 29168 tablet_service.cc:1467] Processing CreateTablet for tablet 96fb0fc5919d42d7bedd770945935b42 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.796219 29167 tablet_service.cc:1467] Processing CreateTablet for tablet 078ffca14316466aa3906d2ff0567c70 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:06.796474 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fb0fc5919d42d7bedd770945935b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.796597 30408 tablet_bootstrap.cc:654] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.796660 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078ffca14316466aa3906d2ff0567c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:06.797107 30409 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:06.798715 30409 tablet_bootstrap.cc:654] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.809854 30412 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:06.813670 30412 tablet_bootstrap.cc:654] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.815200 30409 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:06.815389 30409 ts_tablet_manager.cc:1367] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240502 14:16:06.816174 30409 raft_consensus.cc:348] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:06.817622 30409 raft_consensus.cc:374] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.817747 30409 raft_consensus.cc:729] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.818053 30409 consensus_queue.cc:260] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:06.818665 30409 ts_tablet_manager.cc:1398] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:06.819644 30408 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:06.819825 30408 ts_tablet_manager.cc:1367] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.008s
I20240502 14:16:06.820617 30408 raft_consensus.cc:348] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:06.820772 30407 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:06.820914 30408 raft_consensus.cc:374] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.821053 30408 raft_consensus.cc:729] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.821327 30408 consensus_queue.cc:260] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:06.822800 30408 ts_tablet_manager.cc:1398] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:16:06.822899 30407 tablet_bootstrap.cc:654] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.823145 30409 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:06.823412 30408 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:06.824815 30409 tablet_bootstrap.cc:654] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.826090 30408 tablet_bootstrap.cc:654] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.829504 30412 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:06.829666 30412 ts_tablet_manager.cc:1367] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 14:16:06.830353 30412 raft_consensus.cc:348] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.831526 30412 raft_consensus.cc:374] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.831635 30412 raft_consensus.cc:729] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.831900 30412 consensus_queue.cc:260] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.832459 30412 ts_tablet_manager.cc:1398] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:06.832799 30412 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:06.835131 30412 tablet_bootstrap.cc:654] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.837136 30407 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:06.837353 30407 ts_tablet_manager.cc:1367] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240502 14:16:06.838053 30407 raft_consensus.cc:348] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:06.838200 30407 raft_consensus.cc:374] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.838284 30407 raft_consensus.cc:729] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.838480 30407 consensus_queue.cc:260] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:06.840596 30407 ts_tablet_manager.cc:1398] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:16:06.840855 30407 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:06.842396 30407 tablet_bootstrap.cc:654] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.844249 30412 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:06.844403 30412 ts_tablet_manager.cc:1367] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240502 14:16:06.844456 30409 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:06.844678 30409 ts_tablet_manager.cc:1367] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20240502 14:16:06.844996 30408 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:06.845203 30408 ts_tablet_manager.cc:1367] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.006s
I20240502 14:16:06.845232 30407 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:06.845278 30412 raft_consensus.cc:348] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.845466 30407 ts_tablet_manager.cc:1367] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 14:16:06.845547 30412 raft_consensus.cc:374] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.845496 30409 raft_consensus.cc:348] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.845665 30412 raft_consensus.cc:729] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.845741 30409 raft_consensus.cc:374] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.845870 30409 raft_consensus.cc:729] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.846066 30412 consensus_queue.cc:260] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.846135 30409 consensus_queue.cc:260] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.846191 30408 raft_consensus.cc:348] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.846690 30409 ts_tablet_manager.cc:1398] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:16:06.846742 30412 ts_tablet_manager.cc:1398] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:16:06.846319 30407 raft_consensus.cc:348] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.847165 30407 raft_consensus.cc:374] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.847183 30408 raft_consensus.cc:374] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.847275 30408 raft_consensus.cc:729] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.847316 30407 raft_consensus.cc:729] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.847589 30408 consensus_queue.cc:260] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.848239 30408 ts_tablet_manager.cc:1398] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:16:06.848524 30407 consensus_queue.cc:260] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.849273 30407 ts_tablet_manager.cc:1398] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:16:06.849642 30407 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:06.852031 30407 tablet_bootstrap.cc:654] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:06.853986 30407 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:06.854104 30407 ts_tablet_manager.cc:1367] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:16:06.854620 30407 raft_consensus.cc:348] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.854782 30407 raft_consensus.cc:374] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:06.854877 30407 raft_consensus.cc:729] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:06.855057 30407 consensus_queue.cc:260] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.855548 30407 ts_tablet_manager.cc:1398] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:16:06.910445 30206 raft_consensus.cc:482] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:06.910601 30206 raft_consensus.cc:504] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.911255 30206 leader_election.cc:288] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:06.911583 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ffca14316466aa3906d2ff0567c70" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:06.911795 29451 raft_consensus.cc:2454] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:06.911710 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ffca14316466aa3906d2ff0567c70" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:06.911962 29189 raft_consensus.cc:2454] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:06.912114 29256 leader_election.cc:302] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters:
I20240502 14:16:06.912351 30206 raft_consensus.cc:2781] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:06.912447 30206 raft_consensus.cc:482] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:06.912556 30206 raft_consensus.cc:3037] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.914431 30206 raft_consensus.cc:504] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.914954 30206 leader_election.cc:288] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Requested vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:06.915324 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ffca14316466aa3906d2ff0567c70" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:06.915421 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ffca14316466aa3906d2ff0567c70" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:06.915500 29451 raft_consensus.cc:3037] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.915620 29189 raft_consensus.cc:3037] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.917222 29451 raft_consensus.cc:2454] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:06.917393 29189 raft_consensus.cc:2454] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:06.917587 29256 leader_election.cc:302] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters:
I20240502 14:16:06.917814 30206 raft_consensus.cc:2781] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:06.917935 30206 raft_consensus.cc:686] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Running, Role: LEADER
I20240502 14:16:06.918211 30206 consensus_queue.cc:237] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:06.920903 30344 raft_consensus.cc:482] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:06.921065 30344 raft_consensus.cc:504] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.920975 28913 catalog_manager.cc:5484] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:06.921725 30344 leader_election.cc:288] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:06.922053 29320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fb0fc5919d42d7bedd770945935b42" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:06.922266 29320 raft_consensus.cc:2454] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:06.923252 28990 leader_election.cc:302] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:06.923556 30344 raft_consensus.cc:2781] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:06.923642 30344 raft_consensus.cc:482] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:06.923756 30344 raft_consensus.cc:3037] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.923882 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fb0fc5919d42d7bedd770945935b42" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:06.924072 29189 raft_consensus.cc:2454] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:06.925782 30344 raft_consensus.cc:504] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.926311 30344 leader_election.cc:288] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:06.926563 29320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fb0fc5919d42d7bedd770945935b42" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:06.926746 29320 raft_consensus.cc:3037] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.926704 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fb0fc5919d42d7bedd770945935b42" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:06.926878 29189 raft_consensus.cc:3037] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:06.929508 29320 raft_consensus.cc:2454] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:06.929522 29189 raft_consensus.cc:2454] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:06.929878 28990 leader_election.cc:302] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:06.930110 30344 raft_consensus.cc:2781] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:06.930258 30344 raft_consensus.cc:686] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:06.930644 30344 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:06.933218 28913 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:06.999305 30350 raft_consensus.cc:482] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:06.999477 30350 raft_consensus.cc:504] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:07.000057 30350 leader_election.cc:288] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:07.000528 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302978e1f6034b3693b3a0abacdc494e" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:07.000559 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302978e1f6034b3693b3a0abacdc494e" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:07.000753 29189 raft_consensus.cc:2454] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:07.000831 29057 raft_consensus.cc:2454] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:07.001178 29388 leader_election.cc:302] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:07.001482 30350 raft_consensus.cc:2781] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:07.001610 30350 raft_consensus.cc:482] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:07.001720 30350 raft_consensus.cc:3037] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:07.003666 30350 raft_consensus.cc:504] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:07.004285 30350 leader_election.cc:288] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:07.004626 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302978e1f6034b3693b3a0abacdc494e" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:07.004742 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302978e1f6034b3693b3a0abacdc494e" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:07.004850 29189 raft_consensus.cc:3037] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:07.004957 29057 raft_consensus.cc:3037] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:07.007690 29057 raft_consensus.cc:2454] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:07.007690 29189 raft_consensus.cc:2454] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:07.008080 29387 leader_election.cc:302] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:07.008354 30350 raft_consensus.cc:2781] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:07.008488 30350 raft_consensus.cc:686] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:07.008783 30350 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:07.011307 28913 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:07.306046 30206 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:07.312096 30339 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:07.369858 30423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:07.370151 30423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:07.377496 30423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:16:07.409117 30350 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:07.415414 30331 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:07.455479 30344 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:07.461335 30344 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:07.613837 29385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.67:40039, user_credentials={real_user=slave}} blocked reactor thread for 71699.1us
INSERT report
rows total: 2000
time total: 258.306 ms
time per row: 0.129153 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:16:08.687496 28915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:08.696579 29035 tablet_service.cc:1467] Processing CreateTablet for tablet d67ee220c86f4fc7bc1db8ae08d668d1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.696709 29432 tablet_service.cc:1467] Processing CreateTablet for tablet d67ee220c86f4fc7bc1db8ae08d668d1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.697080 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67ee220c86f4fc7bc1db8ae08d668d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.697077 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67ee220c86f4fc7bc1db8ae08d668d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.697149 29036 tablet_service.cc:1467] Processing CreateTablet for tablet 366dbe98fccd42cb8e369876230984fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.697543 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366dbe98fccd42cb8e369876230984fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.698462 29299 tablet_service.cc:1467] Processing CreateTablet for tablet d67ee220c86f4fc7bc1db8ae08d668d1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.698849 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67ee220c86f4fc7bc1db8ae08d668d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.701071 29037 tablet_service.cc:1467] Processing CreateTablet for tablet d95b84899c994d45a569d5a4dda124ed (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.701423 29037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95b84899c994d45a569d5a4dda124ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.702553 29300 tablet_service.cc:1467] Processing CreateTablet for tablet 366dbe98fccd42cb8e369876230984fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.703056 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366dbe98fccd42cb8e369876230984fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.705595 29431 tablet_service.cc:1467] Processing CreateTablet for tablet d95b84899c994d45a569d5a4dda124ed (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.705965 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95b84899c994d45a569d5a4dda124ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.707502 29167 tablet_service.cc:1467] Processing CreateTablet for tablet d95b84899c994d45a569d5a4dda124ed (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.707926 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95b84899c994d45a569d5a4dda124ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.711366 30494 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:08.711444 29430 tablet_service.cc:1467] Processing CreateTablet for tablet 366dbe98fccd42cb8e369876230984fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:08.711794 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366dbe98fccd42cb8e369876230984fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.713018 30494 tablet_bootstrap.cc:654] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.713845 30492 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:08.716115 30492 tablet_bootstrap.cc:654] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.718633 30494 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:08.718808 30494 ts_tablet_manager.cc:1367] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:16:08.720331 30494 raft_consensus.cc:348] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.720619 30494 raft_consensus.cc:374] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.720741 30494 raft_consensus.cc:729] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.722410 30494 consensus_queue.cc:260] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.724160 30492 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:08.724360 30492 ts_tablet_manager.cc:1367] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240502 14:16:08.725178 30492 raft_consensus.cc:348] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.725431 30492 raft_consensus.cc:374] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.725594 30492 raft_consensus.cc:729] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.725878 30492 consensus_queue.cc:260] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.726485 30492 ts_tablet_manager.cc:1398] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:08.726785 30492 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:08.728837 30492 tablet_bootstrap.cc:654] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.732167 30493 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:08.732728 30498 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:08.733755 30492 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:08.733778 30493 tablet_bootstrap.cc:654] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.733978 30492 ts_tablet_manager.cc:1367] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 14:16:08.735066 30492 raft_consensus.cc:348] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:08.735342 30492 raft_consensus.cc:374] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.735469 30492 raft_consensus.cc:729] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.735622 30498 tablet_bootstrap.cc:654] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.735839 30492 consensus_queue.cc:260] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:08.736510 30492 ts_tablet_manager.cc:1398] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:08.736685 30494 ts_tablet_manager.cc:1398] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:16:08.737175 30492 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:08.738656 30492 tablet_bootstrap.cc:654] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.743525 30494 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:08.744205 30498 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:08.744362 30498 ts_tablet_manager.cc:1367] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 14:16:08.745178 30494 tablet_bootstrap.cc:654] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.745177 30498 raft_consensus.cc:348] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:08.745383 30498 raft_consensus.cc:374] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.745483 30498 raft_consensus.cc:729] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.745760 30498 consensus_queue.cc:260] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:08.747102 30498 ts_tablet_manager.cc:1398] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:08.747192 30493 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:08.747337 30493 ts_tablet_manager.cc:1367] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20240502 14:16:08.747797 30492 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:08.747987 30492 ts_tablet_manager.cc:1367] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 14:16:08.748121 30493 raft_consensus.cc:348] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.748325 30493 raft_consensus.cc:374] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.748416 30493 raft_consensus.cc:729] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.748623 30493 consensus_queue.cc:260] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.748769 30492 raft_consensus.cc:348] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.748977 30494 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:08.749117 30494 ts_tablet_manager.cc:1367] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:16:08.749151 30493 ts_tablet_manager.cc:1398] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:08.749482 30493 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:08.749255 30492 raft_consensus.cc:374] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.749816 30494 raft_consensus.cc:348] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:08.750041 30494 raft_consensus.cc:374] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.750044 30492 raft_consensus.cc:729] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.750106 30494 raft_consensus.cc:729] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.750330 30492 consensus_queue.cc:260] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.750360 30494 consensus_queue.cc:260] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:08.750862 30494 ts_tablet_manager.cc:1398] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:08.751062 30493 tablet_bootstrap.cc:654] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.751184 30494 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:08.751451 30492 ts_tablet_manager.cc:1398] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:08.753006 30494 tablet_bootstrap.cc:654] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:08.755815 30494 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:08.756016 30494 ts_tablet_manager.cc:1367] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:16:08.757443 30493 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:08.757637 30493 ts_tablet_manager.cc:1367] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240502 14:16:08.757930 30494 raft_consensus.cc:348] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.758145 30494 raft_consensus.cc:374] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.758244 30494 raft_consensus.cc:729] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.758356 30493 raft_consensus.cc:348] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.758462 30494 consensus_queue.cc:260] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.758584 30493 raft_consensus.cc:374] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:08.758673 30493 raft_consensus.cc:729] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.759004 30494 ts_tablet_manager.cc:1398] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:08.758917 30493 consensus_queue.cc:260] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.759454 30493 ts_tablet_manager.cc:1398] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:08.789872 30470 raft_consensus.cc:482] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:08.790071 30470 raft_consensus.cc:504] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.790844 30470 leader_election.cc:288] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:08.791342 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366dbe98fccd42cb8e369876230984fe" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:08.791579 29057 raft_consensus.cc:2454] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:08.791529 29320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366dbe98fccd42cb8e369876230984fe" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:08.791779 29320 raft_consensus.cc:2454] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:08.792256 29385 leader_election.cc:302] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters:
I20240502 14:16:08.792692 30470 raft_consensus.cc:2781] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:08.792820 30470 raft_consensus.cc:482] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:08.792922 30470 raft_consensus.cc:3037] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:08.795814 30470 raft_consensus.cc:504] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.796880 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366dbe98fccd42cb8e369876230984fe" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:08.796924 29320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366dbe98fccd42cb8e369876230984fe" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:08.797072 29057 raft_consensus.cc:3037] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:08.797101 29320 raft_consensus.cc:3037] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:08.799863 29057 raft_consensus.cc:2454] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:08.799919 29320 raft_consensus.cc:2454] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:08.800309 29385 leader_election.cc:302] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters:
I20240502 14:16:08.800603 30470 leader_election.cc:288] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:08.800850 30470 raft_consensus.cc:2781] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:08.801028 30470 raft_consensus.cc:686] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:08.801355 30470 consensus_queue.cc:237] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.808864 28915 catalog_manager.cc:5484] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:08.848585 30339 raft_consensus.cc:482] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:08.848778 30339 raft_consensus.cc:504] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.849359 30339 leader_election.cc:288] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:08.849861 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67ee220c86f4fc7bc1db8ae08d668d1" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:08.849972 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67ee220c86f4fc7bc1db8ae08d668d1" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:08.850091 29057 raft_consensus.cc:2454] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:08.850214 29451 raft_consensus.cc:2454] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:08.850502 29256 leader_election.cc:302] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:08.850780 30339 raft_consensus.cc:2781] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:08.850894 30339 raft_consensus.cc:482] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:08.850979 30339 raft_consensus.cc:3037] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:08.852947 30339 raft_consensus.cc:504] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.853474 30339 leader_election.cc:288] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Requested vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:08.853871 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67ee220c86f4fc7bc1db8ae08d668d1" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:08.854045 29057 raft_consensus.cc:3037] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:08.853964 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67ee220c86f4fc7bc1db8ae08d668d1" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:08.854187 29451 raft_consensus.cc:3037] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:08.856264 29057 raft_consensus.cc:2454] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:08.856525 29451 raft_consensus.cc:2454] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:08.856601 29256 leader_election.cc:302] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:08.856920 30339 raft_consensus.cc:2781] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:08.857255 30339 raft_consensus.cc:686] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Running, Role: LEADER
I20240502 14:16:08.857498 30339 consensus_queue.cc:237] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:08.860258 28915 catalog_manager.cc:5484] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:09.125991 30470 raft_consensus.cc:482] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:09.126205 30470 raft_consensus.cc:504] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:09.126781 30470 leader_election.cc:288] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:09.127363 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:09.127374 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:09.127606 29189 raft_consensus.cc:2454] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:09.127622 29057 raft_consensus.cc:2454] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:09.128058 29387 leader_election.cc:302] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:09.128340 30470 raft_consensus.cc:2781] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:09.128435 30470 raft_consensus.cc:482] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:09.128548 30470 raft_consensus.cc:3037] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:09.130424 30470 raft_consensus.cc:504] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:09.130983 30470 leader_election.cc:288] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:09.131386 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:09.131430 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:09.131561 29189 raft_consensus.cc:3037] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:09.131589 29057 raft_consensus.cc:3037] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:09.133370 29057 raft_consensus.cc:2454] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:09.133422 29189 raft_consensus.cc:2454] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:09.133806 29387 leader_election.cc:302] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:09.134088 30470 raft_consensus.cc:2781] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:09.134243 30470 raft_consensus.cc:686] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:09.134516 30470 consensus_queue.cc:237] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:09.137176 28915 catalog_manager.cc:5484] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:09.254493 30470 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:09.265553 30505 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:09.299134 30339 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:09.311213 30518 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:09.550704 30510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:09.551028 30510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:09.557739 30510 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:16:09.624812 30513 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:09.629240 30470 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 245.806 ms
time per row: 0.122903 ms
I20240502 14:16:09.952286 28912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:09.960566 29167 tablet_service.cc:1467] Processing CreateTablet for tablet 093e29d30b2646d68e84b8b9ca86ca9e (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.960821 29299 tablet_service.cc:1467] Processing CreateTablet for tablet 093e29d30b2646d68e84b8b9ca86ca9e (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.961011 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e29d30b2646d68e84b8b9ca86ca9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.961241 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e29d30b2646d68e84b8b9ca86ca9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.965487 30575 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:09.965878 29169 tablet_service.cc:1467] Processing CreateTablet for tablet a6eb9dee735a4115874ef93636a559ab (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.965926 29430 tablet_service.cc:1467] Processing CreateTablet for tablet a6eb9dee735a4115874ef93636a559ab (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.966238 29169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb9dee735a4115874ef93636a559ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.966152 29300 tablet_service.cc:1467] Processing CreateTablet for tablet a6eb9dee735a4115874ef93636a559ab (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.966342 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb9dee735a4115874ef93636a559ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.966491 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb9dee735a4115874ef93636a559ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.967381 30575 tablet_bootstrap.cc:654] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:09.967459 29168 tablet_service.cc:1467] Processing CreateTablet for tablet a230503874a348f4a19647179fe25123 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.967828 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a230503874a348f4a19647179fe25123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.971410 29301 tablet_service.cc:1467] Processing CreateTablet for tablet a230503874a348f4a19647179fe25123 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.971772 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a230503874a348f4a19647179fe25123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.971972 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 093e29d30b2646d68e84b8b9ca86ca9e (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.972275 29036 tablet_service.cc:1467] Processing CreateTablet for tablet a230503874a348f4a19647179fe25123 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:09.972397 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e29d30b2646d68e84b8b9ca86ca9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.972651 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a230503874a348f4a19647179fe25123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.989568 30578 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:09.989851 30575 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:09.989058 30579 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:09.990324 30575 ts_tablet_manager.cc:1367] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.010s
I20240502 14:16:09.992166 30579 tablet_bootstrap.cc:654] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:09.992161 30578 tablet_bootstrap.cc:654] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:09.993819 30576 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:09.993834 30575 raft_consensus.cc:348] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:09.994050 30575 raft_consensus.cc:374] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:09.994138 30575 raft_consensus.cc:729] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:09.994405 30575 consensus_queue.cc:260] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:09.995868 30576 tablet_bootstrap.cc:654] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:09.996021 30579 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:09.996075 30578 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:09.996241 30578 ts_tablet_manager.cc:1367] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 14:16:09.997074 30578 raft_consensus.cc:348] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:09.997345 30578 raft_consensus.cc:374] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:09.997294 30579 ts_tablet_manager.cc:1367] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:16:09.997459 30575 ts_tablet_manager.cc:1398] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:16:09.997457 30578 raft_consensus.cc:729] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:09.997713 30578 consensus_queue.cc:260] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:09.998375 30578 ts_tablet_manager.cc:1398] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:09.998353 30579 raft_consensus.cc:348] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:09.998642 30579 raft_consensus.cc:374] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:09.998700 30578 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:09.998809 30579 raft_consensus.cc:729] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:09.999184 30579 consensus_queue.cc:260] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:09.999821 30579 ts_tablet_manager.cc:1398] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:16:10.000483 30578 tablet_bootstrap.cc:654] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:10.001927 30575 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:10.005820 30575 tablet_bootstrap.cc:654] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:10.007429 30575 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:10.007599 30575 ts_tablet_manager.cc:1367] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:16:10.007822 30578 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:10.007972 30578 ts_tablet_manager.cc:1367] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240502 14:16:10.008252 30575 raft_consensus.cc:348] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.008448 30575 raft_consensus.cc:374] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:10.008519 30575 raft_consensus.cc:729] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:10.008534 30578 raft_consensus.cc:348] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.008752 30578 raft_consensus.cc:374] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:10.008821 30578 raft_consensus.cc:729] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:10.008760 30575 consensus_queue.cc:260] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.009057 30578 consensus_queue.cc:260] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.009444 30578 ts_tablet_manager.cc:1398] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:16:10.009552 30455 raft_consensus.cc:482] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:10.009729 30455 raft_consensus.cc:504] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:10.009958 30575 ts_tablet_manager.cc:1398] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:10.010212 30575 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:10.010704 30455 leader_election.cc:288] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:10.011495 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e29d30b2646d68e84b8b9ca86ca9e" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:10.011675 30575 tablet_bootstrap.cc:654] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
W20240502 14:16:10.013132 28990 leader_election.cc:341] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:16:10.016947 30576 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:10.017210 30576 ts_tablet_manager.cc:1367] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.006s
I20240502 14:16:10.018536 30576 raft_consensus.cc:348] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:10.018893 30576 raft_consensus.cc:374] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:10.019119 30576 raft_consensus.cc:729] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:10.019511 30576 consensus_queue.cc:260] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:10.020830 30576 ts_tablet_manager.cc:1398] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:16:10.020952 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e29d30b2646d68e84b8b9ca86ca9e" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:10.021097 30576 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:10.021191 29189 raft_consensus.cc:2454] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:10.021325 30575 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:10.021447 30575 ts_tablet_manager.cc:1367] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 14:16:10.022296 30576 tablet_bootstrap.cc:654] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:10.022176 30575 raft_consensus.cc:348] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:10.022367 30575 raft_consensus.cc:374] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:10.022672 30575 raft_consensus.cc:729] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:10.022950 30575 consensus_queue.cc:260] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:10.023600 30575 ts_tablet_manager.cc:1398] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:10.024761 28993 leader_election.cc:302] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters: 1b3c20f7bfd548769f6bd40148590b74
I20240502 14:16:10.025065 30585 raft_consensus.cc:2781] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:10.025156 30585 raft_consensus.cc:482] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:10.025998 30585 raft_consensus.cc:3037] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.028209 30585 raft_consensus.cc:504] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:10.028761 30585 leader_election.cc:288] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:10.029403 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e29d30b2646d68e84b8b9ca86ca9e" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:10.029549 29189 raft_consensus.cc:3037] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.031373 29189 raft_consensus.cc:2454] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:10.031711 28993 leader_election.cc:302] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9329e8a7a2d1453d90b76b5965315837, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:10.031960 30585 raft_consensus.cc:2781] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:10.031927 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e29d30b2646d68e84b8b9ca86ca9e" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:10.032109 30585 raft_consensus.cc:686] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:10.032148 29321 raft_consensus.cc:3037] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.032405 30585 consensus_queue.cc:237] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:10.033246 30576 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:10.033442 30576 ts_tablet_manager.cc:1367] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240502 14:16:10.034181 30576 raft_consensus.cc:348] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.034348 30576 raft_consensus.cc:374] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:10.034452 30576 raft_consensus.cc:729] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:10.034703 30576 consensus_queue.cc:260] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.035243 30576 ts_tablet_manager.cc:1398] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:10.035635 30576 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:10.035574 29321 raft_consensus.cc:2454] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:10.037364 30576 tablet_bootstrap.cc:654] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:10.044209 30576 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:10.044348 30576 ts_tablet_manager.cc:1367] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240502 14:16:10.045035 30576 raft_consensus.cc:348] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:10.045213 30576 raft_consensus.cc:374] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:10.045306 30576 raft_consensus.cc:729] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:10.045550 30576 consensus_queue.cc:260] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:10.045358 28912 catalog_manager.cc:5484] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:10.046039 30576 ts_tablet_manager.cc:1398] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:16:10.141867 30470 raft_consensus.cc:482] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:10.142099 30470 raft_consensus.cc:504] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:10.142737 30470 leader_election.cc:288] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:10.143296 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb9dee735a4115874ef93636a559ab" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:10.143488 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb9dee735a4115874ef93636a559ab" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:10.143563 29321 raft_consensus.cc:2454] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:10.143728 29189 raft_consensus.cc:2454] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:10.143990 29385 leader_election.cc:302] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74; no voters:
I20240502 14:16:10.144266 30470 raft_consensus.cc:2781] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:10.144363 30470 raft_consensus.cc:482] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:10.144456 30470 raft_consensus.cc:3037] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.146378 30470 raft_consensus.cc:504] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:10.146925 30470 leader_election.cc:288] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:10.147413 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb9dee735a4115874ef93636a559ab" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:10.147419 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eb9dee735a4115874ef93636a559ab" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:10.147598 29189 raft_consensus.cc:3037] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.147603 29321 raft_consensus.cc:3037] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.149609 29189 raft_consensus.cc:2454] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:10.149725 29321 raft_consensus.cc:2454] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:10.150072 29388 leader_election.cc:302] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:10.150353 30470 raft_consensus.cc:2781] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:10.150686 30470 raft_consensus.cc:686] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:10.150938 30470 consensus_queue.cc:237] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:10.154714 28912 catalog_manager.cc:5484] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:10.190557 30585 raft_consensus.cc:482] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:10.190766 30585 raft_consensus.cc:504] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.191433 30585 leader_election.cc:288] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:10.191916 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a230503874a348f4a19647179fe25123" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:10.191910 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a230503874a348f4a19647179fe25123" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:10.192152 29321 raft_consensus.cc:2454] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:10.192157 29189 raft_consensus.cc:2454] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:10.192541 28990 leader_election.cc:302] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:10.192838 30585 raft_consensus.cc:2781] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:10.192939 30585 raft_consensus.cc:482] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:10.193033 30585 raft_consensus.cc:3037] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.194960 30585 raft_consensus.cc:504] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.195467 30585 leader_election.cc:288] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:10.195884 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a230503874a348f4a19647179fe25123" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:10.196051 29321 raft_consensus.cc:3037] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.196040 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a230503874a348f4a19647179fe25123" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:10.196238 29189 raft_consensus.cc:3037] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:10.197852 29321 raft_consensus.cc:2454] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:10.198031 29189 raft_consensus.cc:2454] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:10.198199 28990 leader_election.cc:302] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:10.198467 30585 raft_consensus.cc:2781] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:10.198617 30585 raft_consensus.cc:686] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:10.198873 30585 consensus_queue.cc:237] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:10.201830 28912 catalog_manager.cc:5484] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:10.560104 30585 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:10.566737 30585 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:10.568231 30593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:10.568647 30593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:10.577286 30593 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 14:16:10.686422 29321 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:10.686961 29189 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:10.687621 30513 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:10.688138 30513 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:10.694041 29320 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:10.695437 30600 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:10.701740 29187 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:10.703606 30585 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:10.733773 30631 mvcc.cc:205] Tried to move back new op lower bound from 7023244782343450624 to 7023244780336570368. Current Snapshot: MvccSnapshot[applied={T|T < 7023244782343450624}]
I20240502 14:16:10.739334 30634 mvcc.cc:205] Tried to move back new op lower bound from 7023244782321766400 to 7023244780140056576. Current Snapshot: MvccSnapshot[applied={T|T < 7023244782321766400}]
I20240502 14:16:10.749276 30602 mvcc.cc:205] Tried to move back new op lower bound from 7023244782343450624 to 7023244780336570368. Current Snapshot: MvccSnapshot[applied={T|T < 7023244782343450624}]
INSERT report
rows total: 2000
time total: 184.482 ms
time per row: 0.0922408 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:16:11.838603 28914 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:11.846660 29167 tablet_service.cc:1467] Processing CreateTablet for tablet 39835ac7b2bb4c0a8c4b43b010b09706 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.846932 29430 tablet_service.cc:1467] Processing CreateTablet for tablet 39835ac7b2bb4c0a8c4b43b010b09706 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.847188 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39835ac7b2bb4c0a8c4b43b010b09706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.847369 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39835ac7b2bb4c0a8c4b43b010b09706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.847935 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 39835ac7b2bb4c0a8c4b43b010b09706 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.848089 29432 tablet_service.cc:1467] Processing CreateTablet for tablet e89c7a017200449abcd5d9b9c2bcc912 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.848403 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39835ac7b2bb4c0a8c4b43b010b09706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.848812 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89c7a017200449abcd5d9b9c2bcc912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.852111 29299 tablet_service.cc:1467] Processing CreateTablet for tablet e89c7a017200449abcd5d9b9c2bcc912 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.852497 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89c7a017200449abcd5d9b9c2bcc912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.856470 29036 tablet_service.cc:1467] Processing CreateTablet for tablet b7bfc3bb327b4407ae1aa7742f4b18de (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.856936 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7bfc3bb327b4407ae1aa7742f4b18de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.857455 30661 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:11.857587 29168 tablet_service.cc:1467] Processing CreateTablet for tablet e89c7a017200449abcd5d9b9c2bcc912 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.857971 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89c7a017200449abcd5d9b9c2bcc912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.859194 30661 tablet_bootstrap.cc:654] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.859669 29431 tablet_service.cc:1467] Processing CreateTablet for tablet b7bfc3bb327b4407ae1aa7742f4b18de (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.860023 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7bfc3bb327b4407ae1aa7742f4b18de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.870579 30661 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:11.870708 30664 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:11.871687 29300 tablet_service.cc:1467] Processing CreateTablet for tablet b7bfc3bb327b4407ae1aa7742f4b18de (DEFAULT_TABLE table=rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:11.872092 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7bfc3bb327b4407ae1aa7742f4b18de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:11.873092 30664 tablet_bootstrap.cc:654] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.875461 30663 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:11.876931 30663 tablet_bootstrap.cc:654] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.877028 30664 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:11.877171 30664 ts_tablet_manager.cc:1367] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:16:11.877890 30664 raft_consensus.cc:348] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.878191 30664 raft_consensus.cc:374] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.878473 30664 raft_consensus.cc:729] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.878090 30660 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:11.878800 30661 ts_tablet_manager.cc:1367] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.007s
I20240502 14:16:11.878733 30664 consensus_queue.cc:260] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.879348 30664 ts_tablet_manager.cc:1398] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 14:16:11.879590 30661 raft_consensus.cc:348] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:11.879804 30661 raft_consensus.cc:374] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.879945 30661 raft_consensus.cc:729] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.880082 30664 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:11.880208 30661 consensus_queue.cc:260] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:11.881508 30661 ts_tablet_manager.cc:1398] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:16:11.881736 30664 tablet_bootstrap.cc:654] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.881816 30661 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:11.884675 30661 tablet_bootstrap.cc:654] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.886301 30660 tablet_bootstrap.cc:654] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.888149 30663 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:11.888285 30663 ts_tablet_manager.cc:1367] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:16:11.888900 30663 raft_consensus.cc:348] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.889109 30663 raft_consensus.cc:374] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.889230 30663 raft_consensus.cc:729] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.889427 30663 consensus_queue.cc:260] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.889780 30664 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:11.889999 30664 ts_tablet_manager.cc:1367] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 14:16:11.890005 30663 ts_tablet_manager.cc:1398] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:11.890219 30663 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:11.890889 30664 raft_consensus.cc:348] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:11.891233 30664 raft_consensus.cc:374] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.891361 30664 raft_consensus.cc:729] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.891687 30663 tablet_bootstrap.cc:654] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.891716 30664 consensus_queue.cc:260] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:11.893870 30664 ts_tablet_manager.cc:1398] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:16:11.894960 30660 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:11.895144 30660 ts_tablet_manager.cc:1367] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 14:16:11.896111 30660 raft_consensus.cc:348] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:11.896358 30660 raft_consensus.cc:374] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.896485 30660 raft_consensus.cc:729] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.896550 30663 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:11.896695 30663 ts_tablet_manager.cc:1367] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:16:11.896999 30660 consensus_queue.cc:260] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:11.897624 30660 ts_tablet_manager.cc:1398] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:11.897904 30660 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:11.898010 30663 raft_consensus.cc:348] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.898192 30663 raft_consensus.cc:374] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.898270 30663 raft_consensus.cc:729] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.898545 30663 consensus_queue.cc:260] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.899149 30663 ts_tablet_manager.cc:1398] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:11.900393 30660 tablet_bootstrap.cc:654] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.900434 30661 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:11.900638 30661 ts_tablet_manager.cc:1367] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.009s
I20240502 14:16:11.901402 30661 raft_consensus.cc:348] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.901660 30661 raft_consensus.cc:374] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.901751 30661 raft_consensus.cc:729] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.901986 30661 consensus_queue.cc:260] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.902617 30661 ts_tablet_manager.cc:1398] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:16:11.903685 30660 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:11.903853 30660 ts_tablet_manager.cc:1367] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:16:11.904603 30660 raft_consensus.cc:348] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.904815 30660 raft_consensus.cc:374] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.904896 30660 raft_consensus.cc:729] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.905614 30660 consensus_queue.cc:260] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.906186 30660 ts_tablet_manager.cc:1398] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:11.906456 30660 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:11.907797 30660 tablet_bootstrap.cc:654] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:11.909255 30660 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:11.909373 30660 ts_tablet_manager.cc:1367] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:11.909860 30660 raft_consensus.cc:348] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.910005 30660 raft_consensus.cc:374] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:11.910094 30660 raft_consensus.cc:729] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:11.910291 30660 consensus_queue.cc:260] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.910694 30660 ts_tablet_manager.cc:1398] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:16:11.960775 30590 raft_consensus.cc:482] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:11.960942 30590 raft_consensus.cc:504] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.961585 30590 leader_election.cc:288] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:11.962054 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89c7a017200449abcd5d9b9c2bcc912" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:11.962121 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89c7a017200449abcd5d9b9c2bcc912" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:11.962244 29321 raft_consensus.cc:2454] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:11.962394 29451 raft_consensus.cc:2454] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 0.
I20240502 14:16:11.962703 29121 leader_election.cc:302] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:11.962982 30590 raft_consensus.cc:2781] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:11.963116 30590 raft_consensus.cc:482] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:11.963194 30590 raft_consensus.cc:3037] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:11.965059 30590 raft_consensus.cc:504] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.965584 30590 leader_election.cc:288] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:11.965974 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89c7a017200449abcd5d9b9c2bcc912" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:11.966059 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89c7a017200449abcd5d9b9c2bcc912" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:11.966146 29321 raft_consensus.cc:3037] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:11.966280 29451 raft_consensus.cc:3037] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:11.968272 29321 raft_consensus.cc:2454] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:11.968328 29451 raft_consensus.cc:2454] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 1.
I20240502 14:16:11.968760 29121 leader_election.cc:302] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:11.969053 30590 raft_consensus.cc:2781] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:11.969414 30590 raft_consensus.cc:686] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Becoming Leader. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Running, Role: LEADER
I20240502 14:16:11.969672 30590 consensus_queue.cc:237] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:11.972440 28914 catalog_manager.cc:5484] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbabfd51061b4e509d907a3622febd2e (127.21.58.66). New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:11.985255 30455 raft_consensus.cc:482] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:11.985471 30455 raft_consensus.cc:504] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.986168 30455 leader_election.cc:288] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:11.986490 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bfc3bb327b4407ae1aa7742f4b18de" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:11.986694 29451 raft_consensus.cc:2454] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:11.986682 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bfc3bb327b4407ae1aa7742f4b18de" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:11.986917 29321 raft_consensus.cc:2454] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:11.987044 28992 leader_election.cc:302] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:11.987339 30455 raft_consensus.cc:2781] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:11.987480 30455 raft_consensus.cc:482] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:11.987591 30455 raft_consensus.cc:3037] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:11.989451 30455 raft_consensus.cc:504] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.989975 30455 leader_election.cc:288] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:11.990327 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bfc3bb327b4407ae1aa7742f4b18de" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:11.990386 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bfc3bb327b4407ae1aa7742f4b18de" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:11.990523 29451 raft_consensus.cc:3037] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:11.990576 29321 raft_consensus.cc:3037] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:11.993229 29321 raft_consensus.cc:2454] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:11.993240 29451 raft_consensus.cc:2454] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:11.993602 28992 leader_election.cc:302] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:11.993865 30455 raft_consensus.cc:2781] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:11.994032 30455 raft_consensus.cc:686] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:11.994277 30455 consensus_queue.cc:237] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:11.996901 28914 catalog_manager.cc:5484] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:12.112211 30455 raft_consensus.cc:482] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:12.112390 30455 raft_consensus.cc:504] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:12.112988 30455 leader_election.cc:288] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:12.113451 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39835ac7b2bb4c0a8c4b43b010b09706" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:12.113454 29188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39835ac7b2bb4c0a8c4b43b010b09706" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:12.113692 29188 raft_consensus.cc:2454] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:12.113732 29451 raft_consensus.cc:2454] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:12.114084 28992 leader_election.cc:302] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:12.114351 30455 raft_consensus.cc:2781] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:12.114444 30455 raft_consensus.cc:482] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:12.114544 30455 raft_consensus.cc:3037] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:12.116632 30455 raft_consensus.cc:504] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:12.117091 30455 leader_election.cc:288] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:12.117473 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39835ac7b2bb4c0a8c4b43b010b09706" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:12.117483 29188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39835ac7b2bb4c0a8c4b43b010b09706" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:12.117668 29188 raft_consensus.cc:3037] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:12.117676 29451 raft_consensus.cc:3037] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:12.119527 29451 raft_consensus.cc:2454] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:12.119626 29188 raft_consensus.cc:2454] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:12.119869 28992 leader_election.cc:302] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:12.120157 30455 raft_consensus.cc:2781] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:12.120280 30455 raft_consensus.cc:686] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:12.120520 30455 consensus_queue.cc:237] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:12.123318 28914 catalog_manager.cc:5484] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:12.487349 30455 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:12.495293 30678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:12.495729 30678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:16:12.496585 30455 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:12.507159 30678 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:12.537639 30590 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:12.538468 30590 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 14:16:12.670668 29450 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:12.671900 29188 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:12.672166 30686 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:12.673137 30675 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:12.784711 30725 mvcc.cc:205] Tried to move back new op lower bound from 7023244790451388416 to 7023244788207304704. Current Snapshot: MvccSnapshot[applied={T|T < 7023244790451388416}]
W20240502 14:16:12.807613 28993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.66:44753, user_credentials={real_user=slave}} blocked reactor thread for 71951.4us
INSERT report
rows total: 2000
time total: 260.977 ms
time per row: 0.130488 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:16:13.856782 28907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:13.864998 29168 tablet_service.cc:1467] Processing CreateTablet for tablet c4818cd7d9b44111a04a0d32a4580cdf (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.865257 29036 tablet_service.cc:1467] Processing CreateTablet for tablet c4818cd7d9b44111a04a0d32a4580cdf (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.865468 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4818cd7d9b44111a04a0d32a4580cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.865445 29300 tablet_service.cc:1467] Processing CreateTablet for tablet c4818cd7d9b44111a04a0d32a4580cdf (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.865629 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4818cd7d9b44111a04a0d32a4580cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.865883 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4818cd7d9b44111a04a0d32a4580cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.869244 29430 tablet_service.cc:1467] Processing CreateTablet for tablet 63690fff28c84670b15ed02ba1356e33 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.869376 29432 tablet_service.cc:1467] Processing CreateTablet for tablet d3221d1a1b244a27abc9ee26e0a6edb1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.869659 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63690fff28c84670b15ed02ba1356e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.869602 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 63690fff28c84670b15ed02ba1356e33 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.869881 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3221d1a1b244a27abc9ee26e0a6edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.869947 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63690fff28c84670b15ed02ba1356e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.869858 29299 tablet_service.cc:1467] Processing CreateTablet for tablet d3221d1a1b244a27abc9ee26e0a6edb1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.870225 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3221d1a1b244a27abc9ee26e0a6edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.876082 29169 tablet_service.cc:1467] Processing CreateTablet for tablet d3221d1a1b244a27abc9ee26e0a6edb1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.876320 29167 tablet_service.cc:1467] Processing CreateTablet for tablet 63690fff28c84670b15ed02ba1356e33 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:13.876466 29169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3221d1a1b244a27abc9ee26e0a6edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.876690 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63690fff28c84670b15ed02ba1356e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:13.884253 30749 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:13.887109 30750 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:13.887630 30749 tablet_bootstrap.cc:654] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.889365 30750 tablet_bootstrap.cc:654] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.890669 30751 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:13.893664 30751 tablet_bootstrap.cc:654] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.894424 30748 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:13.895359 30749 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:13.895519 30749 ts_tablet_manager.cc:1367] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 14:16:13.896261 30750 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:13.896493 30750 ts_tablet_manager.cc:1367] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20240502 14:16:13.896629 30748 tablet_bootstrap.cc:654] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.896495 30749 raft_consensus.cc:348] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.896908 30749 raft_consensus.cc:374] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.897080 30749 raft_consensus.cc:729] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.897325 30750 raft_consensus.cc:348] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.897401 30749 consensus_queue.cc:260] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.897580 30750 raft_consensus.cc:374] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.897670 30750 raft_consensus.cc:729] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.898077 30749 ts_tablet_manager.cc:1398] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:13.898023 30750 consensus_queue.cc:260] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.898780 30750 ts_tablet_manager.cc:1398] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:16:13.898917 30749 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:13.899212 30750 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:13.900527 30749 tablet_bootstrap.cc:654] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.901199 30750 tablet_bootstrap.cc:654] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.901834 30748 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:13.901978 30748 ts_tablet_manager.cc:1367] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:16:13.903195 30748 raft_consensus.cc:348] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.903530 30748 raft_consensus.cc:374] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.903695 30748 raft_consensus.cc:729] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.904019 30748 consensus_queue.cc:260] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.906072 30748 ts_tablet_manager.cc:1398] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:16:13.906467 30748 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:13.909150 30750 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:13.909364 30750 ts_tablet_manager.cc:1367] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 14:16:13.909590 30748 tablet_bootstrap.cc:654] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.910181 30750 raft_consensus.cc:348] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.910454 30750 raft_consensus.cc:374] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.910576 30750 raft_consensus.cc:729] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.910890 30750 consensus_queue.cc:260] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.911592 30750 ts_tablet_manager.cc:1398] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:13.911800 30507 raft_consensus.cc:482] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:13.911943 30507 raft_consensus.cc:504] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.912596 30507 leader_election.cc:288] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:13.912706 30751 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:13.912864 30751 ts_tablet_manager.cc:1367] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240502 14:16:13.913014 29188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4818cd7d9b44111a04a0d32a4580cdf" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:13.913168 29056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4818cd7d9b44111a04a0d32a4580cdf" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:13.913342 29056 raft_consensus.cc:2454] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:13.913805 29256 leader_election.cc:302] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:13.914122 30507 raft_consensus.cc:2781] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:13.914235 30507 raft_consensus.cc:482] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:13.914331 30507 raft_consensus.cc:3037] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:16:13.914500 29257 leader_election.cc:341] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:16:13.914958 30751 raft_consensus.cc:348] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.915185 30751 raft_consensus.cc:374] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.915287 30751 raft_consensus.cc:729] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.915491 30751 consensus_queue.cc:260] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.916064 30751 ts_tablet_manager.cc:1398] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:13.916762 30507 raft_consensus.cc:504] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.916893 30751 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:13.917850 29056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4818cd7d9b44111a04a0d32a4580cdf" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:13.918031 29056 raft_consensus.cc:3037] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:13.918576 29188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4818cd7d9b44111a04a0d32a4580cdf" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:13.918974 29188 raft_consensus.cc:3037] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:13.919791 30749 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:13.919955 30749 ts_tablet_manager.cc:1367] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.008s
I20240502 14:16:13.920539 29056 raft_consensus.cc:2454] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:13.920770 30507 leader_election.cc:288] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:13.920684 30749 raft_consensus.cc:348] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.920897 30749 raft_consensus.cc:374] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.920959 29188 raft_consensus.cc:2454] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:13.920995 30749 raft_consensus.cc:729] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.921010 29256 leader_election.cc:302] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:13.921285 30507 raft_consensus.cc:2781] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:13.921272 30749 consensus_queue.cc:260] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.921898 30749 ts_tablet_manager.cc:1398] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:13.922721 30751 tablet_bootstrap.cc:654] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.927572 30748 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:13.927644 30681 raft_consensus.cc:482] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:13.927922 30748 ts_tablet_manager.cc:1367] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20240502 14:16:13.927839 30681 raft_consensus.cc:504] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.928664 30681 leader_election.cc:288] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:13.929117 30748 raft_consensus.cc:348] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.929469 30748 raft_consensus.cc:374] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.929431 29056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63690fff28c84670b15ed02ba1356e33" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:13.929610 30748 raft_consensus.cc:729] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.929726 29056 raft_consensus.cc:2454] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:13.930004 30748 consensus_queue.cc:260] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.930449 29387 leader_election.cc:302] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:13.933109 30681 raft_consensus.cc:2781] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:13.933275 30681 raft_consensus.cc:482] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:13.933382 30681 raft_consensus.cc:3037] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:13.933948 30507 raft_consensus.cc:686] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Running, Role: LEADER
I20240502 14:16:13.934226 30507 consensus_queue.cc:237] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } }
I20240502 14:16:13.939536 30681 raft_consensus.cc:504] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.940886 29056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63690fff28c84670b15ed02ba1356e33" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:13.941141 29056 raft_consensus.cc:3037] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:13.944563 29056 raft_consensus.cc:2454] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:13.945144 29387 leader_election.cc:302] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:13.945576 30681 leader_election.cc:288] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:13.945878 30681 raft_consensus.cc:2781] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:13.946041 30681 raft_consensus.cc:686] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:13.946434 30681 consensus_queue.cc:237] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.952898 28911 catalog_manager.cc:5484] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:13.955325 28907 catalog_manager.cc:5484] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:13.956292 30748 ts_tablet_manager.cc:1398] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.028s user 0.012s sys 0.014s
I20240502 14:16:13.957237 29188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63690fff28c84670b15ed02ba1356e33" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:13.957209 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63690fff28c84670b15ed02ba1356e33" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
W20240502 14:16:13.957906 29388 leader_election.cc:341] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:16:13.958093 30751 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
W20240502 14:16:13.958274 29388 leader_election.cc:341] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:16:13.958287 30751 ts_tablet_manager.cc:1367] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.006s
I20240502 14:16:13.958931 30751 raft_consensus.cc:348] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.959151 30751 raft_consensus.cc:374] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.959228 30751 raft_consensus.cc:729] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.959476 30751 consensus_queue.cc:260] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.959944 30751 ts_tablet_manager.cc:1398] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:13.960306 30751 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:13.961409 30751 tablet_bootstrap.cc:654] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:13.963366 30751 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:13.963485 30751 ts_tablet_manager.cc:1367] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:13.964033 30751 raft_consensus.cc:348] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.964176 30751 raft_consensus.cc:374] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:13.964283 30751 raft_consensus.cc:729] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:13.964497 30751 consensus_queue.cc:260] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:13.964923 30751 ts_tablet_manager.cc:1398] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:16:13.989318 30761 raft_consensus.cc:482] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:13.989507 30761 raft_consensus.cc:504] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.990201 30761 leader_election.cc:288] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:13.990579 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3221d1a1b244a27abc9ee26e0a6edb1" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:13.990577 29320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3221d1a1b244a27abc9ee26e0a6edb1" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:13.990796 29189 raft_consensus.cc:2454] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:13.990788 29320 raft_consensus.cc:2454] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 0.
I20240502 14:16:13.991218 29388 leader_election.cc:302] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 1b3c20f7bfd548769f6bd40148590b74, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:13.991472 30761 raft_consensus.cc:2781] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:13.991561 30761 raft_consensus.cc:482] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:13.991638 30761 raft_consensus.cc:3037] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:13.993410 30761 raft_consensus.cc:504] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:13.993954 30761 leader_election.cc:288] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:13.994367 29320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3221d1a1b244a27abc9ee26e0a6edb1" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:13.994405 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3221d1a1b244a27abc9ee26e0a6edb1" candidate_uuid: "139e612d62744c02bcb5653b7aa1dc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:13.994565 29189 raft_consensus.cc:3037] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:13.994566 29320 raft_consensus.cc:3037] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:13.996368 29189 raft_consensus.cc:2454] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:13.996378 29320 raft_consensus.cc:2454] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139e612d62744c02bcb5653b7aa1dc29 in term 1.
I20240502 14:16:13.996726 29388 leader_election.cc:302] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:13.997084 30761 raft_consensus.cc:2781] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:13.997215 30761 raft_consensus.cc:686] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Leader. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:13.997483 30761 consensus_queue.cc:237] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } }
I20240502 14:16:14.000312 28911 catalog_manager.cc:5484] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68). New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:14.054008 30696 raft_consensus.cc:482] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:14.054190 30696 raft_consensus.cc:504] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.054916 30696 leader_election.cc:288] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:14.055351 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63690fff28c84670b15ed02ba1356e33" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:14.055554 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63690fff28c84670b15ed02ba1356e33" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:14.055675 29057 raft_consensus.cc:2379] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbabfd51061b4e509d907a3622febd2e in current term 1: Already voted for candidate 139e612d62744c02bcb5653b7aa1dc29 in this term.
I20240502 14:16:14.056138 29124 leader_election.cc:302] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbabfd51061b4e509d907a3622febd2e; no voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837
I20240502 14:16:14.056414 30696 raft_consensus.cc:3037] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:14.058247 30696 raft_consensus.cc:2726] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:16:14.331944 30762 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:14.355422 30762 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:14.378690 30761 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:14.378981 30767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:14.379360 30767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:16:14.387565 30780 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:14.388937 30767 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:14.413520 30761 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:14.418670 30761 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 269.918 ms
time per row: 0.134959 ms
I20240502 14:16:14.810215 28911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57244:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:14.821945 29300 tablet_service.cc:1467] Processing CreateTablet for tablet 0272c80ea41149f895074e0e16180b22 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.822085 29432 tablet_service.cc:1467] Processing CreateTablet for tablet 0272c80ea41149f895074e0e16180b22 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.822402 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0272c80ea41149f895074e0e16180b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.822432 29432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0272c80ea41149f895074e0e16180b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.823942 29430 tablet_service.cc:1467] Processing CreateTablet for tablet 432dc9a199294b748414520a2c49a674 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.824270 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432dc9a199294b748414520a2c49a674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.826901 29299 tablet_service.cc:1467] Processing CreateTablet for tablet 432dc9a199294b748414520a2c49a674 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.827314 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432dc9a199294b748414520a2c49a674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.830236 29301 tablet_service.cc:1467] Processing CreateTablet for tablet 3901ce6537fd48ba861fba13767d8aa4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.830631 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901ce6537fd48ba861fba13767d8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.831717 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 432dc9a199294b748414520a2c49a674 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.832134 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432dc9a199294b748414520a2c49a674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.833179 30834 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:14.834458 29167 tablet_service.cc:1467] Processing CreateTablet for tablet 3901ce6537fd48ba861fba13767d8aa4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.835014 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901ce6537fd48ba861fba13767d8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.835562 29036 tablet_service.cc:1467] Processing CreateTablet for tablet 0272c80ea41149f895074e0e16180b22 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.835956 29036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0272c80ea41149f895074e0e16180b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.836628 30835 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:14.837947 30834 tablet_bootstrap.cc:654] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.838339 30835 tablet_bootstrap.cc:654] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.840102 29037 tablet_service.cc:1467] Processing CreateTablet for tablet 3901ce6537fd48ba861fba13767d8aa4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:14.840502 29037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901ce6537fd48ba861fba13767d8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:14.850903 30838 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:14.852880 30838 tablet_bootstrap.cc:654] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.855578 30834 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:14.855816 30834 ts_tablet_manager.cc:1367] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20240502 14:16:14.856909 30834 raft_consensus.cc:348] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.857252 30834 raft_consensus.cc:374] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.857391 30834 raft_consensus.cc:729] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.857681 30834 consensus_queue.cc:260] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.858422 30834 ts_tablet_manager.cc:1398] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:14.859062 30838 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:14.859288 30838 ts_tablet_manager.cc:1367] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 14:16:14.860131 30838 raft_consensus.cc:348] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.860375 30838 raft_consensus.cc:374] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.860527 30838 raft_consensus.cc:729] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.860785 30838 consensus_queue.cc:260] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.862807 30838 ts_tablet_manager.cc:1398] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:14.863132 30838 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:14.864823 30838 tablet_bootstrap.cc:654] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.865015 30834 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:14.865309 30835 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:14.865465 30835 ts_tablet_manager.cc:1367] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.012s
I20240502 14:16:14.866106 30840 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:14.866298 30835 raft_consensus.cc:348] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.866490 30835 raft_consensus.cc:374] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.866580 30835 raft_consensus.cc:729] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.867201 30834 tablet_bootstrap.cc:654] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.867197 30835 consensus_queue.cc:260] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.868008 30840 tablet_bootstrap.cc:654] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.868906 30835 ts_tablet_manager.cc:1398] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:16:14.869201 30835 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:14.870391 30838 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:14.870594 30838 ts_tablet_manager.cc:1367] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 14:16:14.870697 30835 tablet_bootstrap.cc:654] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.871457 30838 raft_consensus.cc:348] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:14.871798 30838 raft_consensus.cc:374] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.871925 30838 raft_consensus.cc:729] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.872284 30838 consensus_queue.cc:260] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:14.873358 30838 ts_tablet_manager.cc:1398] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:14.874115 30834 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: No bootstrap required, opened a new log
I20240502 14:16:14.875317 30834 ts_tablet_manager.cc:1367] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240502 14:16:14.876688 30834 raft_consensus.cc:348] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:14.877027 30834 raft_consensus.cc:374] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.877156 30834 raft_consensus.cc:729] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.878139 30834 consensus_queue.cc:260] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:14.880362 30834 ts_tablet_manager.cc:1398] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:16:14.880949 30838 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:14.897467 30838 tablet_bootstrap.cc:654] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.899986 30838 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: No bootstrap required, opened a new log
I20240502 14:16:14.900182 30838 ts_tablet_manager.cc:1367] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240502 14:16:14.900635 30840 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e: No bootstrap required, opened a new log
I20240502 14:16:14.900635 30835 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:14.900848 30835 ts_tablet_manager.cc:1367] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.008s
I20240502 14:16:14.900866 30840 ts_tablet_manager.cc:1367] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.008s
I20240502 14:16:14.900980 30838 raft_consensus.cc:348] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.901176 30838 raft_consensus.cc:374] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.901284 30838 raft_consensus.cc:729] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.901489 30838 consensus_queue.cc:260] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.901602 30840 raft_consensus.cc:348] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.901574 30835 raft_consensus.cc:348] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:14.901798 30835 raft_consensus.cc:374] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.901811 30840 raft_consensus.cc:374] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.901903 30840 raft_consensus.cc:729] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.901957 30835 raft_consensus.cc:729] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.902101 30840 consensus_queue.cc:260] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.902202 30835 consensus_queue.cc:260] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:14.902801 30838 ts_tablet_manager.cc:1398] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:16:14.904016 30840 ts_tablet_manager.cc:1398] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:14.904417 30835 ts_tablet_manager.cc:1398] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:16:14.905633 30835 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: Bootstrap starting.
I20240502 14:16:14.908963 30835 tablet_bootstrap.cc:654] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:14.910828 30835 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: No bootstrap required, opened a new log
I20240502 14:16:14.910934 30835 ts_tablet_manager.cc:1367] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 14:16:14.911624 30835 raft_consensus.cc:348] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.911829 30835 raft_consensus.cc:374] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:14.911933 30835 raft_consensus.cc:729] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Initialized, Role: FOLLOWER
I20240502 14:16:14.912158 30835 consensus_queue.cc:260] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.912664 30835 ts_tablet_manager.cc:1398] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:16:14.937777 30686 raft_consensus.cc:482] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:14.937971 30686 raft_consensus.cc:504] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.938549 30686 leader_election.cc:288] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:14.938961 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0272c80ea41149f895074e0e16180b22" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:14.939152 29321 raft_consensus.cc:2454] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:14.939060 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0272c80ea41149f895074e0e16180b22" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:14.939322 29450 raft_consensus.cc:2454] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:14.939492 28990 leader_election.cc:302] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:14.939777 30686 raft_consensus.cc:2781] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:14.939875 30686 raft_consensus.cc:482] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:14.939988 30686 raft_consensus.cc:3037] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:14.941780 30686 raft_consensus.cc:504] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.942320 30686 leader_election.cc:288] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:14.942623 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0272c80ea41149f895074e0e16180b22" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:14.942728 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0272c80ea41149f895074e0e16180b22" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:14.942817 29321 raft_consensus.cc:3037] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:14.942898 29450 raft_consensus.cc:3037] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:14.945592 29321 raft_consensus.cc:2454] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:14.945623 29450 raft_consensus.cc:2454] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:14.945926 28992 leader_election.cc:302] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:14.946208 30686 raft_consensus.cc:2781] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:14.946550 30686 raft_consensus.cc:686] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:14.946805 30686 consensus_queue.cc:237] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:14.949746 28910 catalog_manager.cc:5484] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:15.032011 30762 raft_consensus.cc:482] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:15.032200 30762 raft_consensus.cc:504] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:15.032869 30762 leader_election.cc:288] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:15.033301 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432dc9a199294b748414520a2c49a674" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837" is_pre_election: true
I20240502 14:16:15.033516 29057 raft_consensus.cc:2454] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:15.033473 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432dc9a199294b748414520a2c49a674" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29" is_pre_election: true
I20240502 14:16:15.033768 29450 raft_consensus.cc:2454] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 0.
I20240502 14:16:15.033946 29256 leader_election.cc:302] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:15.034209 30762 raft_consensus.cc:2781] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:15.034327 30762 raft_consensus.cc:482] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:15.034415 30762 raft_consensus.cc:3037] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:15.036316 30762 raft_consensus.cc:504] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:15.036787 30762 leader_election.cc:288] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Requested vote from peers 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:15.037127 29057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432dc9a199294b748414520a2c49a674" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9329e8a7a2d1453d90b76b5965315837"
I20240502 14:16:15.037227 29450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432dc9a199294b748414520a2c49a674" candidate_uuid: "1b3c20f7bfd548769f6bd40148590b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:15.037307 29057 raft_consensus.cc:3037] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:15.037420 29450 raft_consensus.cc:3037] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:15.039054 29057 raft_consensus.cc:2454] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:15.039244 29450 raft_consensus.cc:2454] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3c20f7bfd548769f6bd40148590b74 in term 1.
I20240502 14:16:15.039471 29256 leader_election.cc:302] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:15.039724 30762 raft_consensus.cc:2781] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:15.039906 30762 raft_consensus.cc:686] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3c20f7bfd548769f6bd40148590b74, State: Running, Role: LEADER
I20240502 14:16:15.040151 30762 consensus_queue.cc:237] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } }
I20240502 14:16:15.041471 30686 raft_consensus.cc:482] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:16:15.041595 30686 raft_consensus.cc:504] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:15.042124 30686 leader_election.cc:288] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:15.042481 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901ce6537fd48ba861fba13767d8aa4" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74" is_pre_election: true
I20240502 14:16:15.042606 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901ce6537fd48ba861fba13767d8aa4" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e" is_pre_election: true
I20240502 14:16:15.042682 29321 raft_consensus.cc:2454] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:15.042874 29189 raft_consensus.cc:2454] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 0.
I20240502 14:16:15.043156 28990 leader_election.cc:302] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:15.043498 30686 raft_consensus.cc:2781] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:16:15.043586 30686 raft_consensus.cc:482] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:16:15.043670 30686 raft_consensus.cc:3037] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:15.043614 28910 catalog_manager.cc:5484] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:15.045670 30686 raft_consensus.cc:504] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:15.046238 30686 leader_election.cc:288] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:15.046563 29321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901ce6537fd48ba861fba13767d8aa4" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:15.046783 29321 raft_consensus.cc:3037] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:15.046850 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901ce6537fd48ba861fba13767d8aa4" candidate_uuid: "9329e8a7a2d1453d90b76b5965315837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:15.047063 29189 raft_consensus.cc:3037] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:15.049039 29321 raft_consensus.cc:2454] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:15.049444 28990 leader_election.cc:302] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3c20f7bfd548769f6bd40148590b74, 9329e8a7a2d1453d90b76b5965315837; no voters:
I20240502 14:16:15.049654 29189 raft_consensus.cc:2454] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9329e8a7a2d1453d90b76b5965315837 in term 1.
I20240502 14:16:15.049683 30686 raft_consensus.cc:2781] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:15.049793 30686 raft_consensus.cc:686] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Becoming Leader. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Running, Role: LEADER
I20240502 14:16:15.050036 30686 consensus_queue.cc:237] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } }
I20240502 14:16:15.052647 28910 catalog_manager.cc:5484] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } }
W20240502 14:16:15.433835 30851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:15.434121 30851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:15.441336 30851 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:15.457949 30686 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:15.463476 30847 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240502 14:16:15.514469 30866 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:15.545329 30866 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:15.580124 29450 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:15.580703 29057 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:15.581597 30771 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:16:15.582190 30762 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:16:15.730024 28992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.68:46509, user_credentials={real_user=slave}} blocked reactor thread for 56545.7us
INSERT report
rows total: 2000
time total: 243.203 ms
time per row: 0.121601 ms
I20240502 14:16:15.830492 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:36535
--local_ip_for_outbound_sockets=127.21.58.69
--tserver_master_addrs=127.21.58.126:42921
--webserver_port=46643
--webserver_interface=127.21.58.69
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:16:16.170017 30909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:16.170260 30909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:16.170339 30909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:16.178858 30909 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:16.178972 30909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:16.179116 30909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:16:16.188480 30909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:36535
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=46643
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:16.189723 30909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:16.191457 30909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:16.203272 30914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:16.203982 30915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:16.204123 30917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:16.204797 30909 server_base.cc:1034] running on GCE node
I20240502 14:16:16.205327 30909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:16.206218 30909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:16.207413 30909 hybrid_clock.cc:648] HybridClock initialized: now 1714659376207349 us; error 49 us; skew 500 ppm
I20240502 14:16:16.207774 30909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:16.210494 30909 webserver.cc:460] Webserver started at http://127.21.58.69:46643/ using document root <none> and password file <none>
I20240502 14:16:16.211356 30909 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:16.211485 30909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:16.216858 30909 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:16:16.219636 30923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:16.220793 30909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:16.220966 30909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ede5e1f11f254824a335e1e43f40b6d1"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:16:16.221700 30909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:16.245362 30909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:16.246142 30909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:16.246426 30909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:16.247361 30909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:16.249023 30909 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:16.249109 30909 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:16.249222 30909 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:16.249269 30909 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:16.285130 30909 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:36535
I20240502 14:16:16.285171 31036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:36535 every 8 connection(s)
I20240502 14:16:16.286784 30909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:16:16.290447 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 30909
I20240502 14:16:16.291165 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.70:41819
--local_ip_for_outbound_sockets=127.21.58.70
--tserver_master_addrs=127.21.58.126:42921
--webserver_port=34405
--webserver_interface=127.21.58.70
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:16.299566 31038 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:16:16.299831 31038 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:16.300752 31038 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:16:16.302590 28914 ts_manager.cc:194] Re-registered known tserver with Master: ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535)
I20240502 14:16:16.303815 28914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:55421
W20240502 14:16:16.626092 31042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:16.626394 31042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:16.626521 31042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:16.633589 31042 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:16.633713 31042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:16.633888 31042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.70
I20240502 14:16:16.642540 31042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.70:41819
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.58.70
--webserver_port=34405
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.70
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:16.643812 31042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:16.645438 31042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:16.657155 31052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:16.657294 31050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:16.657895 31049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:16.658476 31042 server_base.cc:1034] running on GCE node
I20240502 14:16:16.659018 31042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:16.659833 31042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:16.661017 31042 hybrid_clock.cc:648] HybridClock initialized: now 1714659376660976 us; error 30 us; skew 500 ppm
I20240502 14:16:16.661340 31042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:16.663452 31042 webserver.cc:460] Webserver started at http://127.21.58.70:34405/ using document root <none> and password file <none>
I20240502 14:16:16.664324 31042 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:16.664474 31042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:16.669682 31042 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:16:16.672410 31058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:16.673666 31042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 14:16:16.673815 31042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3bb852e0a21446bbbe824bdefb16036c"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:16:16.674614 31042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:16.687211 31042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:16.687938 31042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:16.688191 31042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:16.689028 31042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:16.690479 31042 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:16.690544 31042 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:16.690654 31042 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:16.690697 31042 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:16.732703 31042 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.70:41819
I20240502 14:16:16.732740 31171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.70:41819 every 8 connection(s)
I20240502 14:16:16.734165 31042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:16:16.744349 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 31042
I20240502 14:16:16.745159 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.71:44703
--local_ip_for_outbound_sockets=127.21.58.71
--tserver_master_addrs=127.21.58.126:42921
--webserver_port=36361
--webserver_interface=127.21.58.71
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:16.747375 31172 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:16:16.747690 31172 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:16.748669 31172 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:16:16.750201 28914 ts_manager.cc:194] Re-registered known tserver with Master: 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819)
I20240502 14:16:16.751317 28914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.70:33419
W20240502 14:16:17.085773 31175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:17.086078 31175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:17.086215 31175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:17.093586 31175 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:17.093708 31175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:17.093905 31175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.71
I20240502 14:16:17.103163 31175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.71:44703
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.58.71
--webserver_port=36361
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.71
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:17.104908 31175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:17.106511 31175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:17.118084 31181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:17.118575 31184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:17.119164 31182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:17.119433 31175 server_base.cc:1034] running on GCE node
I20240502 14:16:17.120030 31175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:17.120908 31175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:17.122084 31175 hybrid_clock.cc:648] HybridClock initialized: now 1714659377122043 us; error 30 us; skew 500 ppm
I20240502 14:16:17.122368 31175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:17.124410 31175 webserver.cc:460] Webserver started at http://127.21.58.71:36361/ using document root <none> and password file <none>
I20240502 14:16:17.125102 31175 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:17.125212 31175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:17.130004 31175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:16:17.132539 31190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:17.133760 31175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:17.133920 31175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b7dacf388234446b9a11ce750978ca75"
format_stamp: "Formatted at 2024-05-02 14:15:55 on dist-test-slave-wtgm"
I20240502 14:16:17.134725 31175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:17.153581 31175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:17.154296 31175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:17.154532 31175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:17.155365 31175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:17.156797 31175 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:17.156865 31175 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:17.156955 31175 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:17.156998 31175 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:17.187904 31175 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.71:44703
I20240502 14:16:17.187938 31303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.71:44703 every 8 connection(s)
I20240502 14:16:17.189256 31175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:16:17.195811 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 31175
I20240502 14:16:17.196574 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.72:36549
--local_ip_for_outbound_sockets=127.21.58.72
--tserver_master_addrs=127.21.58.126:42921
--webserver_port=46021
--webserver_interface=127.21.58.72
--builtin_ntp_servers=127.21.58.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:17.201622 31304 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:16:17.201867 31304 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:17.202637 31304 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:16:17.203917 28914 ts_manager.cc:194] Re-registered known tserver with Master: b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703)
I20240502 14:16:17.204756 28914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.71:37921
I20240502 14:16:17.306232 31038 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
W20240502 14:16:17.545701 31307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:17.546028 31307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:17.546168 31307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:17.554183 31307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:17.554306 31307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:17.554474 31307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.72
I20240502 14:16:17.564347 31307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.72:36549
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.58.72
--webserver_port=46021
--tserver_master_addrs=127.21.58.126:42921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.72
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:17.565768 31307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:17.567716 31307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:17.578943 31313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:17.579010 31316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:17.579581 31314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:17.580335 31307 server_base.cc:1034] running on GCE node
I20240502 14:16:17.580884 31307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:17.581673 31307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:17.582854 31307 hybrid_clock.cc:648] HybridClock initialized: now 1714659377582813 us; error 31 us; skew 500 ppm
I20240502 14:16:17.583168 31307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:17.589751 31307 webserver.cc:460] Webserver started at http://127.21.58.72:46021/ using document root <none> and password file <none>
I20240502 14:16:17.590417 31307 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:17.590525 31307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:17.595489 31307 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:16:17.597988 31322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:17.599159 31307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:17.599292 31307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d653f68cb53b4ed79d5e688987ed8d4c"
format_stamp: "Formatted at 2024-05-02 14:15:56 on dist-test-slave-wtgm"
I20240502 14:16:17.600030 31307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:17.620546 31307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:17.621351 31307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:17.621637 31307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:17.622457 31307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:17.623829 31307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:17.623896 31307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:17.623984 31307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:17.624027 31307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:17.655314 31307 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.72:36549
I20240502 14:16:17.655354 31435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.72:36549 every 8 connection(s)
I20240502 14:16:17.656724 31307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:16:17.658071 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 31307
I20240502 14:16:17.672821 31436 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42921
I20240502 14:16:17.673121 31436 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:17.673916 31436 heartbeater.cc:511] Master 127.21.58.126:42921 requested a full tablet report, sending...
I20240502 14:16:17.675176 28914 ts_manager.cc:194] Re-registered known tserver with Master: d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549)
I20240502 14:16:17.675912 28914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.72:59123
I20240502 14:16:17.753595 31172 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
I20240502 14:16:18.210016 31304 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
I20240502 14:16:18.678102 31436 heartbeater.cc:503] Master 127.21.58.126:42921 was elected leader, sending a full tablet report...
I20240502 14:16:18.774170 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:18.805192 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:18.856176 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:18.984445 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:18.995292 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:18.997660 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.012961 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:19.025151 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.026201 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:19.027256 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.050546 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:19.073603 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.093209 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.094421 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.135080 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:19.174098 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.285265 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:19.301894 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.308275 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:19.330122 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.334218 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.371091 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:19.375996 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:19.383913 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.902724 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.904345 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:19.915370 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:19.916321 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.919553 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:19.962136 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:20.012521 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.014225 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:20.166698 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:20.171171 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.176133 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.198822 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.202179 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:20.208763 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:20.231148 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:20.234970 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.270677 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.300040 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:20.346532 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:20.356178 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:20.361229 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.395040 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:20.396093 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.411969 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.694530 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:20.694891 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:20.748306 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.751314 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:20.752981 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.776054 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.854182 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:20.860360 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:20.964128 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:21.033406 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:21.091053 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.140163 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:21.146689 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.170886 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:21.183002 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.206029 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.550743 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:21.601601 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.633553 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.634047 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:21.646243 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:21.646755 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:21.651580 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:21.700053 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:21.731936 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.758972 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.778275 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.780229 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.796000 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.852330 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:21.867165 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:21.875273 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.896260 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:21.915304 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:21.926541 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.935024 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.958679 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:21.983949 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:22.008476 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:22.019286 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.363607 29057 consensus_queue.cc:237] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:22.390771 29451 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:22.393075 32015 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:22.422086 29189 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:22.422784 32015 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:22.431931 32010 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:22.444391 29189 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:22.454491 28910 catalog_manager.cc:5484] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:22.460130 29450 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
W20240502 14:16:22.489670 28993 consensus_peers.cc:485] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fde0741532a4067a4806f4bc0b0d465. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:22.552213 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:22.564554 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:22.583843 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:22.602576 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:22.629629 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:22.630313 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.635679 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.673534 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.677106 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:22.700235 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.708477 29057 consensus_queue.cc:237] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:22.726917 29189 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:22.727722 29320 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:22.727705 32085 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:22.731220 32085 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:22.745039 32083 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:22.748551 29188 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:22.758785 28911 catalog_manager.cc:5484] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:22.783383 29321 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:22.799196 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:16:22.813818 28992 consensus_peers.cc:485] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3901ce6537fd48ba861fba13767d8aa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:22.841238 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:22.846383 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.872609 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.884124 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:22.884779 29321 consensus_queue.cc:237] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:22.886785 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:22.936483 32159 ts_tablet_manager.cc:902] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:22.967890 29189 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:22.968566 29057 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:22.968621 32127 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:22.969338 32127 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:22.980157 32127 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:22.983729 29188 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:22.989514 28909 catalog_manager.cc:5484] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:22.993986 32159 tablet_copy_client.cc:308] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:22.999507 29056 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:23.034672 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 3fde0741532a4067a4806f4bc0b0d465 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:39523)
I20240502 14:16:23.035099 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 3fde0741532a4067a4806f4bc0b0d465 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:39523: session id = d653f68cb53b4ed79d5e688987ed8d4c-3fde0741532a4067a4806f4bc0b0d465
W20240502 14:16:23.041394 29254 consensus_peers.cc:485] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f17889f58a34bc0afa40a246e5debc6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.044510 29077 tablet_copy_source_session.cc:205] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:23.052327 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fde0741532a4067a4806f4bc0b0d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:23.071468 32159 tablet_copy_client.cc:783] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:23.071991 32159 tablet_copy_client.cc:647] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:23.075148 32159 tablet_copy_client.cc:523] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:23.089502 32159 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:23.099606 29451 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:23.105674 32159 log.cc:824] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:23.119478 29321 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.119546 29188 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.120127 31595 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.120783 31595 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.153496 32201 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:23.156627 29321 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:23.164150 28911 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
W20240502 14:16:23.176267 29386 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6a0a881fcec43b291148f071e4474fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.171021 29188 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:23.205657 29188 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:23.239271 29056 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.240052 31452 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.263060 29451 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.263970 31452 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.287504 29450 consensus_queue.cc:237] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:23.321012 32235 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:23.328063 32159 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:23.331843 32159 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:23.334421 32159 ts_tablet_manager.cc:1367] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.245s user 0.080s sys 0.016s
I20240502 14:16:23.335492 32231 ts_tablet_manager.cc:902] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:23.342928 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:16:23.361464 29123 consensus_peers.cc:485] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 981ec6067d624100bddba4383b417c50. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.379729 29321 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.380717 32201 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.382608 29189 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.385582 32201 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.408295 32201 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:23.411615 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.413733 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:23.416432 29320 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
W20240502 14:16:23.419006 29388 consensus_peers.cc:485] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6eb9dee735a4115874ef93636a559ab. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.423441 28910 catalog_manager.cc:5484] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:23.432575 29189 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:23.447994 29056 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:23.452749 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.354877 32159 raft_consensus.cc:348] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:23.453862 32159 raft_consensus.cc:729] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:23.454255 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.454478 32159 consensus_queue.cc:260] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:23.456175 29450 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:23.457319 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.463575 28910 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:23.487962 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:23.501575 32231 tablet_copy_client.cc:308] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:23.502918 32273 ts_tablet_manager.cc:902] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:23.533835 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:23.538228 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:23.542198 29321 consensus_queue.cc:237] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
W20240502 14:16:23.550674 29254 consensus_peers.cc:485] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 432dc9a199294b748414520a2c49a674. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.550976 29056 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.551517 29450 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.552094 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:23.553071 31044 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.553560 31044 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.556996 32273 tablet_copy_client.cc:308] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:23.563663 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.569854 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.574862 32159 ts_tablet_manager.cc:1398] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.240s user 0.009s sys 0.005s
I20240502 14:16:23.576385 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-3fde0741532a4067a4806f4bc0b0d465 received from {username='slave'} at 127.21.58.72:39523
I20240502 14:16:23.576545 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-3fde0741532a4067a4806f4bc0b0d465 on tablet 3fde0741532a4067a4806f4bc0b0d465 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:23.606170 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 3901ce6537fd48ba861fba13767d8aa4 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:60009)
I20240502 14:16:23.606366 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 3901ce6537fd48ba861fba13767d8aa4 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:60009: session id = 3bb852e0a21446bbbe824bdefb16036c-3901ce6537fd48ba861fba13767d8aa4
I20240502 14:16:23.609449 29077 tablet_copy_source_session.cc:205] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:23.615484 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901ce6537fd48ba861fba13767d8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:23.646288 31044 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:23.651834 29451 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:23.658771 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.661268 28910 catalog_manager.cc:5484] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:23.663623 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:23.669246 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:23.694360 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:23.704381 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet 1f17889f58a34bc0afa40a246e5debc6 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:36693)
I20240502 14:16:23.704674 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet 1f17889f58a34bc0afa40a246e5debc6 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:36693: session id = ede5e1f11f254824a335e1e43f40b6d1-1f17889f58a34bc0afa40a246e5debc6
I20240502 14:16:23.708853 29341 tablet_copy_source_session.cc:205] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:23.714601 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f17889f58a34bc0afa40a246e5debc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:23.765303 32305 ts_tablet_manager.cc:902] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:23.766705 32306 ts_tablet_manager.cc:902] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
W20240502 14:16:23.768288 29256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.65:46783, user_credentials={real_user=slave}} blocked reactor thread for 111274us
I20240502 14:16:23.771324 32159 ts_tablet_manager.cc:902] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:23.775851 29057 consensus_queue.cc:237] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:23.781404 29056 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:23.786928 29189 consensus_queue.cc:237] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:23.810853 29321 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.811826 32234 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:23.818696 29124 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44d94461e22c4ae182afa2a15f3db198. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.819865 29450 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.820591 32234 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.829407 29320 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.830144 32083 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:23.831938 28993 consensus_peers.cc:485] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 093e29d30b2646d68e84b8b9ca86ca9e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.832953 32234 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:23.836704 29189 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.837617 32083 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.837746 29451 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:23.841465 29319 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:23.845448 28910 catalog_manager.cc:5484] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:23.851413 32273 tablet_copy_client.cc:783] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:23.851770 32273 tablet_copy_client.cc:647] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:23.858462 32273 tablet_copy_client.cc:523] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:23.862962 32273 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:23.864624 32083 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:23.870005 29189 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:23.879720 28913 catalog_manager.cc:5484] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:23.881289 32231 tablet_copy_client.cc:783] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:23.883312 32231 tablet_copy_client.cc:647] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:23.895853 32231 tablet_copy_client.cc:523] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:23.911590 29321 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:23.921927 32231 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:23.925287 29450 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:23.926260 32306 tablet_copy_client.cc:308] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:23.950246 29189 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.950340 29320 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:23.951018 31595 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.951478 32201 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:23.960363 31595 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:23.967295 32305 tablet_copy_client.cc:308] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:23.972117 29321 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:23.973124 28913 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:23.974059 32336 ts_tablet_manager.cc:902] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:23.975567 31391 raft_consensus.cc:1203] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 14:16:23.980783 29387 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0fcde0e7cbe41f1a521e669b798aa99. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:23.986567 29189 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:23.995568 32336 tablet_copy_client.cc:308] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:23.996321 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet 432dc9a199294b748414520a2c49a674 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:36693)
I20240502 14:16:23.996484 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet 432dc9a199294b748414520a2c49a674 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:36693: session id = ede5e1f11f254824a335e1e43f40b6d1-432dc9a199294b748414520a2c49a674
I20240502 14:16:23.999402 29341 tablet_copy_source_session.cc:205] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.001209 32336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432dc9a199294b748414520a2c49a674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.012022 32159 tablet_copy_client.cc:308] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:24.034294 29057 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:24.034693 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet e6a0a881fcec43b291148f071e4474fc from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:54103)
I20240502 14:16:24.034989 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet e6a0a881fcec43b291148f071e4474fc from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:54103: session id = b7dacf388234446b9a11ce750978ca75-e6a0a881fcec43b291148f071e4474fc
I20240502 14:16:24.039073 29472 tablet_copy_source_session.cc:205] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.039701 32336 tablet_copy_client.cc:783] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.040957 32336 tablet_copy_client.cc:647] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.044586 32306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a0a881fcec43b291148f071e4474fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.047446 32336 tablet_copy_client.cc:523] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.052335 32336 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:24.059964 29450 consensus_queue.cc:237] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:24.064671 32306 tablet_copy_client.cc:783] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.064827 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet a6eb9dee735a4115874ef93636a559ab from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:60507)
I20240502 14:16:24.064991 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet a6eb9dee735a4115874ef93636a559ab from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:60507: session id = d653f68cb53b4ed79d5e688987ed8d4c-a6eb9dee735a4115874ef93636a559ab
I20240502 14:16:24.065027 32306 tablet_copy_client.cc:647] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.069185 29472 tablet_copy_source_session.cc:205] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.070612 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb9dee735a4115874ef93636a559ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.072386 29056 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.074456 32306 tablet_copy_client.cc:523] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.074501 31595 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.077826 32159 tablet_copy_client.cc:783] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.078109 32159 tablet_copy_client.cc:647] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
W20240502 14:16:24.079828 29385 consensus_peers.cc:485] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63690fff28c84670b15ed02ba1356e33. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.080257 32159 tablet_copy_client.cc:523] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.082666 32306 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:24.082863 32201 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:24.083472 32159 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:24.090175 29055 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:24.091897 28910 catalog_manager.cc:5484] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:24.092962 29321 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.093700 32015 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.100684 32306 log.cc:824] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:24.057183 32273 log.cc:824] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:24.121243 29209 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet 981ec6067d624100bddba4383b417c50 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:53205)
I20240502 14:16:24.121556 29209 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet 981ec6067d624100bddba4383b417c50 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:53205: session id = b7dacf388234446b9a11ce750978ca75-981ec6067d624100bddba4383b417c50
I20240502 14:16:24.121606 29188 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.122241 32015 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.126291 32330 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.126834 29321 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.133168 28913 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
W20240502 14:16:24.137962 28991 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fb0fc5919d42d7bedd770945935b42. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.139560 32231 log.cc:824] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:24.143841 29189 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.144595 31595 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.172101 29209 tablet_copy_source_session.cc:205] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.174299 32305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981ec6067d624100bddba4383b417c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.181382 32305 tablet_copy_client.cc:783] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.181686 32305 tablet_copy_client.cc:647] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.191174 32305 tablet_copy_client.cc:523] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.194876 32305 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:24.211373 29189 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:24.216329 29188 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.232769 32231 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:24.240922 32231 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:24.242484 32336 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:24.243049 32336 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:24.243621 32336 ts_tablet_manager.cc:1367] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.191s user 0.055s sys 0.026s
I20240502 14:16:24.250484 32336 raft_consensus.cc:348] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:24.251042 32336 raft_consensus.cc:729] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:24.251703 32336 consensus_queue.cc:260] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:24.256927 32231 ts_tablet_manager.cc:1367] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.335s user 0.065s sys 0.004s
I20240502 14:16:24.262136 32336 ts_tablet_manager.cc:1398] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.018s user 0.009s sys 0.005s
I20240502 14:16:24.263336 29341 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-432dc9a199294b748414520a2c49a674 received from {username='slave'} at 127.21.58.69:36693
I20240502 14:16:24.263489 29341 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-432dc9a199294b748414520a2c49a674 on tablet 432dc9a199294b748414520a2c49a674 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:24.277307 32231 raft_consensus.cc:348] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:24.278750 29450 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:24.281320 29188 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.281971 32201 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.282218 29320 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:16:24.283013 29387 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0fcde0e7cbe41f1a521e669b798aa99. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.283582 31595 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.286716 32201 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:24.292722 28910 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:24.297359 29388 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0fcde0e7cbe41f1a521e669b798aa99. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.298085 29188 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:24.298274 29320 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:24.308802 29188 consensus_queue.cc:237] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:24.309928 32231 raft_consensus.cc:729] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:24.310787 32231 consensus_queue.cc:260] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:24.311843 32273 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:24.321473 32273 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:24.321943 32273 ts_tablet_manager.cc:1367] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.459s user 0.066s sys 0.001s
I20240502 14:16:24.326860 32273 raft_consensus.cc:348] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:24.327337 32273 raft_consensus.cc:729] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:24.327725 32273 consensus_queue.cc:260] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:24.337770 32273 ts_tablet_manager.cc:1398] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:16:24.339226 32231 ts_tablet_manager.cc:1398] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.082s user 0.012s sys 0.000s
I20240502 14:16:24.340344 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-3901ce6537fd48ba861fba13767d8aa4 received from {username='slave'} at 127.21.58.70:60009
I20240502 14:16:24.340520 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-3901ce6537fd48ba861fba13767d8aa4 on tablet 3901ce6537fd48ba861fba13767d8aa4 with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:24.349706 29341 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-1f17889f58a34bc0afa40a246e5debc6 received from {username='slave'} at 127.21.58.69:36693
I20240502 14:16:24.349834 29341 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-1f17889f58a34bc0afa40a246e5debc6 on tablet 1f17889f58a34bc0afa40a246e5debc6 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:24.358136 29320 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.358793 32239 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:24.359614 29123 consensus_peers.cc:485] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e89c7a017200449abcd5d9b9c2bcc912. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.362962 29450 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.364382 32234 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.376492 32234 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.382628 29450 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.383111 32159 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:24.383560 32159 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:24.383831 32159 ts_tablet_manager.cc:1367] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.300s user 0.049s sys 0.029s
I20240502 14:16:24.384475 28910 catalog_manager.cc:5484] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:24.399982 32159 raft_consensus.cc:348] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:24.400334 32159 raft_consensus.cc:729] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:24.400519 32159 consensus_queue.cc:260] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:24.401891 32159 ts_tablet_manager.cc:1398] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20240502 14:16:24.402532 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-a6eb9dee735a4115874ef93636a559ab received from {username='slave'} at 127.21.58.72:60507
I20240502 14:16:24.402666 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-a6eb9dee735a4115874ef93636a559ab on tablet a6eb9dee735a4115874ef93636a559ab with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:24.410339 29188 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:24.412186 32231 ts_tablet_manager.cc:902] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:24.413957 32159 ts_tablet_manager.cc:902] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:24.419961 30992 raft_consensus.cc:1203] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:24.427273 32305 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 14:16:24.429929 29124 consensus_peers.cc:485] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd4e4241d42d483ab6203606a4d92141. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.430665 29321 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.431459 32234 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.432281 32305 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:24.433012 29055 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.433851 32234 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.433890 32305 ts_tablet_manager.cc:1367] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.239s user 0.059s sys 0.002s
I20240502 14:16:24.447106 32159 tablet_copy_client.cc:308] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:24.447889 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 093e29d30b2646d68e84b8b9ca86ca9e from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:39523)
I20240502 14:16:24.448037 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 093e29d30b2646d68e84b8b9ca86ca9e from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:39523: session id = d653f68cb53b4ed79d5e688987ed8d4c-093e29d30b2646d68e84b8b9ca86ca9e
I20240502 14:16:24.451004 29077 tablet_copy_source_session.cc:205] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.452525 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e29d30b2646d68e84b8b9ca86ca9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.461072 32305 raft_consensus.cc:348] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:24.463299 32305 raft_consensus.cc:729] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:24.463874 32273 ts_tablet_manager.cc:902] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:24.464522 32305 consensus_queue.cc:260] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:24.465034 32159 tablet_copy_client.cc:783] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.465418 32159 tablet_copy_client.cc:647] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.467509 32159 tablet_copy_client.cc:523] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.462044 32306 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:24.468400 32306 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:24.468672 32306 ts_tablet_manager.cc:1367] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.386s user 0.060s sys 0.016s
I20240502 14:16:24.469658 32306 raft_consensus.cc:348] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:24.469879 32306 raft_consensus.cc:729] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:24.470139 32306 consensus_queue.cc:260] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:24.479471 32306 ts_tablet_manager.cc:1398] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20240502 14:16:24.482518 32305 ts_tablet_manager.cc:1398] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.037s user 0.005s sys 0.004s
I20240502 14:16:24.483830 29209 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-981ec6067d624100bddba4383b417c50 received from {username='slave'} at 127.21.58.71:53205
I20240502 14:16:24.484012 29209 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session b7dacf388234446b9a11ce750978ca75-981ec6067d624100bddba4383b417c50 on tablet 981ec6067d624100bddba4383b417c50 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:24.486519 32159 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:24.489099 32305 ts_tablet_manager.cc:902] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:24.489966 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-e6a0a881fcec43b291148f071e4474fc received from {username='slave'} at 127.21.58.71:54103
I20240502 14:16:24.490094 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session b7dacf388234446b9a11ce750978ca75-e6a0a881fcec43b291148f071e4474fc on tablet e6a0a881fcec43b291148f071e4474fc with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:24.493001 32234 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:24.495983 29319 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:24.506186 28910 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:24.507350 30992 raft_consensus.cc:1203] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:24.523051 29188 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
W20240502 14:16:24.527300 29123 consensus_peers.cc:485] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd4e4241d42d483ab6203606a4d92141. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:24.528155 29124 consensus_peers.cc:485] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd4e4241d42d483ab6203606a4d92141. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.529284 29319 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.529858 32239 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.540236 29057 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.542304 32234 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.543339 29057 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:24.549675 32239 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.550287 29057 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.557112 28909 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.580802 29321 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.591426 32083 raft_consensus.cc:1050] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:24.592466 32083 consensus_queue.cc:237] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:24.602159 31127 raft_consensus.cc:1203] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:24.615192 29321 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:24.619200 29450 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:24.623811 29057 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:24.624186 29387 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4dbd9bc62484dd1998080d289db098f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.624665 32198 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.630573 29321 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.631410 32198 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.632613 29318 consensus_queue.cc:237] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:24.656589 31391 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.657325 32330 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.663748 32273 tablet_copy_client.cc:308] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:24.664026 32231 tablet_copy_client.cc:308] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:24.664927 31258 raft_consensus.cc:1203] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:24.664911 31257 raft_consensus.cc:1203] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:24.670193 32159 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:24.670794 32159 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:24.671141 32159 ts_tablet_manager.cc:1367] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.185s user 0.070s sys 0.004s
I20240502 14:16:24.672538 32159 raft_consensus.cc:348] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:24.672879 32159 raft_consensus.cc:729] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:24.673172 32159 consensus_queue.cc:260] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:24.674638 32159 ts_tablet_manager.cc:1398] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:24.675441 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-093e29d30b2646d68e84b8b9ca86ca9e received from {username='slave'} at 127.21.58.72:39523
I20240502 14:16:24.675606 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-093e29d30b2646d68e84b8b9ca86ca9e on tablet 093e29d30b2646d68e84b8b9ca86ca9e with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:24.676151 29188 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.676740 32330 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.680729 32201 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:24.685500 29320 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:24.688992 29450 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.689419 28910 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:24.690397 32330 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.691895 29056 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:24.714776 29055 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:24.715180 29254 consensus_peers.cc:485] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d67ee220c86f4fc7bc1db8ae08d668d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:24.715335 32300 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.727809 29451 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:24.729372 32300 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:24.734035 32300 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:24.735986 29056 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:24.749814 32423 ts_tablet_manager.cc:902] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:24.763908 28913 catalog_manager.cc:5484] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:24.786175 32423 tablet_copy_client.cc:308] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:24.787089 29209 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet e89c7a017200449abcd5d9b9c2bcc912 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:53205)
I20240502 14:16:24.787325 29209 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet e89c7a017200449abcd5d9b9c2bcc912 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:53205: session id = b7dacf388234446b9a11ce750978ca75-e89c7a017200449abcd5d9b9c2bcc912
I20240502 14:16:24.787108 32083 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.792672 31391 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.790230 29450 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:24.800083 29209 tablet_copy_source_session.cc:205] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.801651 29188 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.811401 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 63690fff28c84670b15ed02ba1356e33 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:59701)
I20240502 14:16:24.811553 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 63690fff28c84670b15ed02ba1356e33 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:59701: session id = ede5e1f11f254824a335e1e43f40b6d1-63690fff28c84670b15ed02ba1356e33
I20240502 14:16:24.814328 29472 tablet_copy_source_session.cc:205] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.815953 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63690fff28c84670b15ed02ba1356e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.818754 32423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89c7a017200449abcd5d9b9c2bcc912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.821887 32273 tablet_copy_client.cc:783] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.822160 32273 tablet_copy_client.cc:647] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.822474 31044 raft_consensus.cc:1050] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:24.823640 31044 consensus_queue.cc:237] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:24.829576 29209 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet 44d94461e22c4ae182afa2a15f3db198 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:45939)
I20240502 14:16:24.829737 29209 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet 44d94461e22c4ae182afa2a15f3db198 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:45939: session id = 3bb852e0a21446bbbe824bdefb16036c-44d94461e22c4ae182afa2a15f3db198
I20240502 14:16:24.833413 31390 raft_consensus.cc:1203] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:24.837270 29209 tablet_copy_source_session.cc:205] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.837424 28910 catalog_manager.cc:5484] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.840363 32273 tablet_copy_client.cc:523] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.843772 32273 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:24.845355 32479 ts_tablet_manager.cc:902] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:24.847098 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d94461e22c4ae182afa2a15f3db198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.847121 32305 tablet_copy_client.cc:308] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:24.847258 29451 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.854702 32231 tablet_copy_client.cc:783] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.859658 32231 tablet_copy_client.cc:647] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.861359 29189 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.862249 32286 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.864928 30992 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.865550 32286 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.873754 32423 tablet_copy_client.cc:783] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.874027 32423 tablet_copy_client.cc:647] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.874094 29055 consensus_queue.cc:237] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:24.875985 32423 tablet_copy_client.cc:523] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.882520 32423 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:24.882697 32159 ts_tablet_manager.cc:902] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:24.884841 32231 tablet_copy_client.cc:523] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.885788 29451 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:24.888612 32486 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:24.890022 31391 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:24.891592 29055 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.892848 31044 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:24.893844 29451 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.898890 32231 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:24.899298 32015 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:24.899689 28913 catalog_manager.cc:5484] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.900166 32486 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.907225 31391 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:24.916304 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fde0741532a4067a4806f4bc0b0d465 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:24.923142 32300 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:24.928125 30991 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:24.929690 32159 tablet_copy_client.cc:308] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:24.930413 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet a0fcde0e7cbe41f1a521e669b798aa99 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:60507)
I20240502 14:16:24.930590 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet a0fcde0e7cbe41f1a521e669b798aa99 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:60507: session id = d653f68cb53b4ed79d5e688987ed8d4c-a0fcde0e7cbe41f1a521e669b798aa99
I20240502 14:16:24.932507 28908 catalog_manager.cc:5484] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:24.933468 29472 tablet_copy_source_session.cc:205] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.934836 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fcde0e7cbe41f1a521e669b798aa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.935266 29055 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:24.939507 31391 raft_consensus.cc:1203] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:24.941063 29188 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:24.946501 29318 consensus_queue.cc:237] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:24.948092 32479 tablet_copy_client.cc:308] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:24.949008 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 96fb0fc5919d42d7bedd770945935b42 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:35201)
I20240502 14:16:24.949211 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 96fb0fc5919d42d7bedd770945935b42 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:35201: session id = b7dacf388234446b9a11ce750978ca75-96fb0fc5919d42d7bedd770945935b42
I20240502 14:16:24.951184 30992 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:24.951661 31044 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:24.951848 29077 tablet_copy_source_session.cc:205] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.954186 32305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fb0fc5919d42d7bedd770945935b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.958914 32507 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:24.961839 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet a0fcde0e7cbe41f1a521e669b798aa99 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:56885)
I20240502 14:16:24.962014 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet a0fcde0e7cbe41f1a521e669b798aa99 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:56885: session id = 3bb852e0a21446bbbe824bdefb16036c-a0fcde0e7cbe41f1a521e669b798aa99
I20240502 14:16:24.962595 30991 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:24.964170 32159 tablet_copy_client.cc:783] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.965080 32159 tablet_copy_client.cc:647] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:24.966856 32159 tablet_copy_client.cc:523] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:24.966976 29472 tablet_copy_source_session.cc:205] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:24.968012 28899 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f17889f58a34bc0afa40a246e5debc6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:24.970197 32159 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:24.970839 32305 tablet_copy_client.cc:783] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:24.969517 28910 catalog_manager.cc:5484] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:24.972059 29167 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fde0741532a4067a4806f4bc0b0d465 with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:24.975579 32479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fcde0e7cbe41f1a521e669b798aa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:24.976035 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f17889f58a34bc0afa40a246e5debc6 with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:24.990813 29189 consensus_queue.cc:237] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:24.995574 32514 tablet_replica.cc:330] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:24.996155 32514 raft_consensus.cc:2229] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:24.996567 32514 raft_consensus.cc:2258] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:24.999604 29318 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:24.999668 29450 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.000180 32436 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:16:25.000447 29124 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:25.000684 32437 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.004141 29187 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.005297 32507 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.007141 29187 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.012735 32514 ts_tablet_manager.cc:1875] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:25.015678 32305 tablet_copy_client.cc:647] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:25.018162 32305 tablet_copy_client.cc:523] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:25.023291 29451 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:25.023566 32513 tablet_replica.cc:330] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:25.024664 32443 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:25.025768 29452 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:25.032671 28911 catalog_manager.cc:5484] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:25.036707 32513 raft_consensus.cc:2229] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:25.037140 32513 raft_consensus.cc:2258] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:25.040642 32513 ts_tablet_manager.cc:1875] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:25.052824 31258 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:16:25.052863 29387 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6a0a881fcec43b291148f071e4474fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:25.053259 29320 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.053591 32201 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.053799 31595 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.058960 32305 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:25.060631 32083 raft_consensus.cc:1050] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:25.061595 32083 consensus_queue.cc:237] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:25.065704 32201 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.071704 29321 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.077677 28911 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:25.089701 32479 tablet_copy_client.cc:783] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:25.090023 32479 tablet_copy_client.cc:647] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:25.093199 32479 tablet_copy_client.cc:523] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:25.097018 32479 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:25.097154 32514 ts_tablet_manager.cc:1888] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:25.097352 32514 log.cc:1196] T 1f17889f58a34bc0afa40a246e5debc6 P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/1f17889f58a34bc0afa40a246e5debc6
I20240502 14:16:25.098728 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet 1f17889f58a34bc0afa40a246e5debc6 (table rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]) successfully deleted
I20240502 14:16:25.097218 32513 ts_tablet_manager.cc:1888] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:25.101920 32513 log.cc:1196] T 3fde0741532a4067a4806f4bc0b0d465 P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fde0741532a4067a4806f4bc0b0d465
I20240502 14:16:25.105939 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet 3fde0741532a4067a4806f4bc0b0d465 (table rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]) successfully deleted
I20240502 14:16:25.122607 32507 raft_consensus.cc:1050] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:25.123492 32507 consensus_queue.cc:237] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:25.124627 29055 consensus_queue.cc:237] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:25.142771 29320 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:25.149850 29321 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:25.150339 32486 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:25.154130 32443 raft_consensus.cc:1050] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:25.155401 32443 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:25.178154 29187 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:25.178910 29189 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.179476 32486 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:25.180243 32201 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.193473 29189 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.198527 32201 raft_consensus.cc:1050] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:25.199537 32201 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:25.114269 32534 ts_tablet_manager.cc:902] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:25.201438 29188 consensus_queue.cc:237] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:25.203701 29187 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:16:25.206068 29387 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6a0a881fcec43b291148f071e4474fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:25.207170 31257 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.207803 31595 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.208570 32015 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:25.209087 31257 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.209940 32083 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.211401 29187 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:25.199458 31126 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.217726 32486 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.219521 32544 ts_tablet_manager.cc:902] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:25.220304 28911 catalog_manager.cc:5484] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:25.236537 30992 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.237316 31044 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.251165 29055 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.252429 29451 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.253291 32300 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.253661 32542 ts_tablet_manager.cc:902] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:25.253973 32300 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:16:25.258652 29125 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44d94461e22c4ae182afa2a15f3db198. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:25.268129 31595 raft_consensus.cc:1050] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:25.268958 31595 consensus_queue.cc:237] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:25.278501 29451 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.279273 32437 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.279459 31256 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.280088 32437 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.280848 32486 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.284089 32558 ts_tablet_manager.cc:902] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:25.284332 29057 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:25.284896 32437 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:25.287189 31126 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.288462 28913 catalog_manager.cc:5484] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:16:25.293816 29125 consensus_peers.cc:485] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e66f9bb135bf4007b1864b63a389fac3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:25.301594 29185 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.303225 29320 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.306782 29451 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:25.307483 32437 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:25.320024 32437 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:25.321712 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:25.328922 28911 catalog_manager.cc:5484] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:25.340546 29056 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.341619 32231 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:25.342172 32231 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:25.342501 32231 ts_tablet_manager.cc:1367] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.444s user 0.071s sys 0.008s
I20240502 14:16:25.343735 32231 raft_consensus.cc:348] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:25.344064 32231 raft_consensus.cc:729] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:25.344274 32231 consensus_queue.cc:260] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:25.349565 32231 ts_tablet_manager.cc:1398] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:16:25.350250 29209 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-44d94461e22c4ae182afa2a15f3db198 received from {username='slave'} at 127.21.58.70:45939
I20240502 14:16:25.350410 29209 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-44d94461e22c4ae182afa2a15f3db198 on tablet 44d94461e22c4ae182afa2a15f3db198 with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:25.351519 32015 raft_consensus.cc:1050] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:25.352428 32015 consensus_queue.cc:237] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:25.361327 29320 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.361394 29189 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.362003 32552 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.363164 32198 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.362427 29057 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:25.366542 29320 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.372949 32198 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.374454 28911 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.381539 29318 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.381623 32443 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.382520 32553 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.393081 31391 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.393815 32486 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.400450 32273 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:25.400885 32273 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:25.401186 32273 ts_tablet_manager.cc:1367] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.558s user 0.049s sys 0.003s
I20240502 14:16:25.403131 29189 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.403771 32486 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.405493 31255 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.415374 32542 tablet_copy_client.cc:308] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:25.416095 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet f4dbd9bc62484dd1998080d289db098f from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:56885)
I20240502 14:16:25.416276 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet f4dbd9bc62484dd1998080d289db098f from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:56885: session id = 3bb852e0a21446bbbe824bdefb16036c-f4dbd9bc62484dd1998080d289db098f
I20240502 14:16:25.416307 32437 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.419095 29472 tablet_copy_source_session.cc:205] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:25.419873 29054 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.420511 32542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dbd9bc62484dd1998080d289db098f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:25.429059 28908 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:25.429803 32542 tablet_copy_client.cc:783] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:25.430035 32542 tablet_copy_client.cc:647] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:25.435101 29452 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.435472 29185 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:25.438304 31257 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.438951 32436 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.440325 31257 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.446624 32605 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.447345 31258 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.451157 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 981ec6067d624100bddba4383b417c50 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:25.452831 28908 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:25.458845 32544 tablet_copy_client.cc:308] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:25.459507 29209 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet fd4e4241d42d483ab6203606a4d92141 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:53205)
I20240502 14:16:25.459697 29209 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet fd4e4241d42d483ab6203606a4d92141 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:53205: session id = b7dacf388234446b9a11ce750978ca75-fd4e4241d42d483ab6203606a4d92141
I20240502 14:16:25.464223 29209 tablet_copy_source_session.cc:205] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:25.469233 32486 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.470939 32544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4e4241d42d483ab6203606a4d92141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:25.477135 29052 consensus_queue.cc:237] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:25.479816 31126 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.480906 32611 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.484364 32330 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.485883 31126 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.486617 32273 raft_consensus.cc:348] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:25.487725 32273 raft_consensus.cc:729] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:25.487974 32273 consensus_queue.cc:260] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:25.495200 28908 catalog_manager.cc:5484] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.501515 32544 tablet_copy_client.cc:783] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:25.502066 32544 tablet_copy_client.cc:647] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
W20240502 14:16:25.504261 28990 consensus_peers.cc:485] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a230503874a348f4a19647179fe25123. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:25.504735 32544 tablet_copy_client.cc:523] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:25.509198 32544 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
W20240502 14:16:25.511104 29388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.66:44753, user_credentials={real_user=slave}} blocked reactor thread for 145326us
I20240502 14:16:25.511967 29185 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.512290 31389 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.513875 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet 981ec6067d624100bddba4383b417c50 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:25.514293 29185 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.519250 32201 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.519858 32201 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.521255 32558 tablet_copy_client.cc:308] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:25.526846 32542 tablet_copy_client.cc:523] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:25.540158 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet d67ee220c86f4fc7bc1db8ae08d668d1 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:36693)
I20240502 14:16:25.540325 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet d67ee220c86f4fc7bc1db8ae08d668d1 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:36693: session id = ede5e1f11f254824a335e1e43f40b6d1-d67ee220c86f4fc7bc1db8ae08d668d1
I20240502 14:16:25.543817 29185 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.545013 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet 3901ce6537fd48ba861fba13767d8aa4 with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:25.545959 29320 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:25.546823 32611 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.547276 32611 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.551282 31044 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.551689 29451 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:25.555265 31255 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:25.555989 32588 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:25.556109 29318 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:25.556481 29341 tablet_copy_source_session.cc:205] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:25.558027 29318 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:25.558998 29452 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:25.560995 32553 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:25.562420 29450 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.563882 32615 ts_tablet_manager.cc:902] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:25.564285 32552 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.564909 29318 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.570523 32553 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.576503 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3901ce6537fd48ba861fba13767d8aa4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:25.576552 29318 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.578737 29185 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.579509 29188 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.579638 29054 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.583774 28913 catalog_manager.cc:5484] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.584337 28913 catalog_manager.cc:5484] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:25.584873 29056 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.586391 32443 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.588162 28909 catalog_manager.cc:5484] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.586854 31390 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.589787 29054 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.594523 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:25.599503 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:25.599851 31126 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:25.607975 32558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67ee220c86f4fc7bc1db8ae08d668d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:25.610273 32330 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:25.611363 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6a0a881fcec43b291148f071e4474fc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:25.612342 29189 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.605618 30992 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.619544 32273 ts_tablet_manager.cc:1398] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.218s user 0.004s sys 0.000s
I20240502 14:16:25.627107 32534 tablet_copy_client.cc:308] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:25.626766 29318 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.628026 29209 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet fd4e4241d42d483ab6203606a4d92141 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:45939)
I20240502 14:16:25.628218 29209 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet fd4e4241d42d483ab6203606a4d92141 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:45939: session id = 3bb852e0a21446bbbe824bdefb16036c-fd4e4241d42d483ab6203606a4d92141
I20240502 14:16:25.630954 29209 tablet_copy_source_session.cc:205] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:25.632442 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-63690fff28c84670b15ed02ba1356e33 received from {username='slave'} at 127.21.58.69:59701
I20240502 14:16:25.632573 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-63690fff28c84670b15ed02ba1356e33 on tablet 63690fff28c84670b15ed02ba1356e33 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:25.636076 32617 tablet_replica.cc:330] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:25.637076 29188 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:25.636973 29319 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:25.638804 31391 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.640290 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4e4241d42d483ab6203606a4d92141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:25.641613 32627 tablet_replica.cc:330] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:25.642127 32627 raft_consensus.cc:2229] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:25.642443 32627 raft_consensus.cc:2258] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:25.647275 29320 consensus_queue.cc:237] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:25.647955 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:25.652972 30990 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.653782 32300 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.658044 31044 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.659664 32615 tablet_copy_client.cc:308] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:25.659801 30990 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.667361 28899 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 432dc9a199294b748414520a2c49a674 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:25.667665 32273 ts_tablet_manager.cc:902] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:25.670614 28909 catalog_manager.cc:5484] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.675819 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet 432dc9a199294b748414520a2c49a674 with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:25.678439 28913 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.687209 29035 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:25.687896 32641 tablet_replica.cc:330] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:25.688545 32641 raft_consensus.cc:2229] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:25.689335 32641 raft_consensus.cc:2258] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:25.690953 32641 ts_tablet_manager.cc:1875] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:25.692057 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:16:25.693090 32624 log.cc:924] Time spent T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240502 14:16:25.693184 32534 tablet_copy_client.cc:783] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:25.693552 32534 tablet_copy_client.cc:647] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
W20240502 14:16:25.693406 32624 consensus_queue.cc:1159] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1b3c20f7bfd548769f6bd40148590b74" status { last_received { term: 1 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 5 } }
I20240502 14:16:25.695453 32534 tablet_copy_client.cc:523] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:25.695461 32627 ts_tablet_manager.cc:1875] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:25.701977 32534 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:25.636750 29452 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:25.637641 32553 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:25.716001 31255 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.719156 29188 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.649438 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:25.732187 32617 raft_consensus.cc:2229] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:25.738423 32423 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:25.742206 32617 raft_consensus.cc:2258] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:25.705760 29451 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.746214 31044 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.748013 29450 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:25.759438 29450 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:25.761291 32273 tablet_copy_client.cc:308] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:25.765429 29054 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:25.766441 32588 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:25.768276 29387 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302978e1f6034b3693b3a0abacdc494e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:25.777122 29189 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:25.777899 32588 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:25.786571 32641 ts_tablet_manager.cc:1888] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:25.788918 32641 log.cc:1196] T 432dc9a199294b748414520a2c49a674 P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/432dc9a199294b748414520a2c49a674
I20240502 14:16:25.792855 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet 432dc9a199294b748414520a2c49a674 (table rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]) successfully deleted
I20240502 14:16:25.870518 32627 ts_tablet_manager.cc:1888] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:25.873584 29449 consensus_queue.cc:237] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:25.880574 29188 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:25.882017 29449 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:25.883298 32588 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:25.888267 31595 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.891505 32542 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:25.894835 28911 catalog_manager.cc:5484] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:25.895510 32617 ts_tablet_manager.cc:1875] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:25.901325 32558 tablet_copy_client.cc:783] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:25.901582 32558 tablet_copy_client.cc:647] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:25.904243 32534 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:25.904824 32534 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:25.905138 32534 ts_tablet_manager.cc:1367] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.203s user 0.078s sys 0.034s
I20240502 14:16:25.906216 32534 raft_consensus.cc:348] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:25.906543 32534 raft_consensus.cc:729] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:25.906785 32534 consensus_queue.cc:260] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:25.908486 32534 ts_tablet_manager.cc:1398] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:25.909250 29209 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-fd4e4241d42d483ab6203606a4d92141 received from {username='slave'} at 127.21.58.70:45939
I20240502 14:16:25.909407 29209 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-fd4e4241d42d483ab6203606a4d92141 on tablet fd4e4241d42d483ab6203606a4d92141 with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:25.778026 32544 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:25.884665 31255 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:25.910805 32423 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:25.910950 32544 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:25.911131 32423 ts_tablet_manager.cc:1367] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 1.029s user 0.050s sys 0.004s
I20240502 14:16:25.911298 32544 ts_tablet_manager.cc:1367] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.402s user 0.055s sys 0.005s
I20240502 14:16:25.911448 32588 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:25.912016 32423 raft_consensus.cc:348] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:25.912283 32544 raft_consensus.cc:348] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:25.912645 32544 raft_consensus.cc:729] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:25.912892 32544 consensus_queue.cc:260] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:25.913607 32544 ts_tablet_manager.cc:1398] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:25.912284 32423 raft_consensus.cc:729] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:25.915468 32423 consensus_queue.cc:260] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:25.916078 32423 ts_tablet_manager.cc:1398] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:16:25.917106 29209 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-fd4e4241d42d483ab6203606a4d92141 received from {username='slave'} at 127.21.58.71:53205
I20240502 14:16:25.917259 29209 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session b7dacf388234446b9a11ce750978ca75-fd4e4241d42d483ab6203606a4d92141 on tablet fd4e4241d42d483ab6203606a4d92141 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:25.918359 32588 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.919528 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-e89c7a017200449abcd5d9b9c2bcc912 received from {username='slave'} at 127.21.58.71:53205
I20240502 14:16:25.919646 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session b7dacf388234446b9a11ce750978ca75-e89c7a017200449abcd5d9b9c2bcc912 on tablet e89c7a017200449abcd5d9b9c2bcc912 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:25.924721 31255 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:25.930738 29188 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:25.941366 28910 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:25.943392 32617 ts_tablet_manager.cc:1888] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:25.943648 32617 log.cc:1196] T 981ec6067d624100bddba4383b417c50 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/981ec6067d624100bddba4383b417c50
I20240502 14:16:25.945412 29167 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a0a881fcec43b291148f071e4474fc with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:25.966044 32558 tablet_copy_client.cc:523] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:25.969705 32201 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.970682 32658 tablet_replica.cc:330] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:25.970969 32658 raft_consensus.cc:2229] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:25.971446 32658 raft_consensus.cc:2258] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:25.975565 29056 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.977213 29188 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:25.979727 32658 ts_tablet_manager.cc:1875] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:25.985500 28911 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:25.993502 29387 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.000945 29054 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.001466 32588 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.006634 32653 ts_tablet_manager.cc:902] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:26.006517 31595 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.009092 31391 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.011029 29056 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.015834 32553 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.017694 31390 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
W20240502 14:16:26.027552 29388 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4dbd9bc62484dd1998080d289db098f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.031654 29452 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.036090 28910 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:26.039964 30992 raft_consensus.cc:1203] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:26.042829 32658 ts_tablet_manager.cc:1888] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:16:26.059929 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:26.065270 32305 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:26.060505 29208 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet 44d94461e22c4ae182afa2a15f3db198 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:49699)
I20240502 14:16:26.076586 29208 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet 44d94461e22c4ae182afa2a15f3db198 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:49699: session id = d653f68cb53b4ed79d5e688987ed8d4c-44d94461e22c4ae182afa2a15f3db198
I20240502 14:16:25.903489 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eb9dee735a4115874ef93636a559ab with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:26.045635 29314 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.046200 29316 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:26.079926 29208 tablet_copy_source_session.cc:205] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.081458 32615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d94461e22c4ae182afa2a15f3db198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.084342 32588 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.084885 32588 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:26.087294 29314 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.088999 32615 tablet_copy_client.cc:783] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.089285 32615 tablet_copy_client.cc:647] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.091014 32615 tablet_copy_client.cc:523] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.095024 32615 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:26.099265 32658 log.cc:1196] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/e6a0a881fcec43b291148f071e4474fc
I20240502 14:16:26.098716 29316 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.101204 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet e6a0a881fcec43b291148f071e4474fc (table rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]) successfully deleted
I20240502 14:16:26.069648 31255 raft_consensus.cc:1203] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:26.097591 31257 raft_consensus.cc:1203] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:16:26.105793 32627 log.cc:1196] T 3901ce6537fd48ba861fba13767d8aa4 P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/3901ce6537fd48ba861fba13767d8aa4
I20240502 14:16:26.106937 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet 3901ce6537fd48ba861fba13767d8aa4 (table rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]) successfully deleted
I20240502 14:16:26.107610 32627 tablet_replica.cc:330] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:26.118026 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6eb9dee735a4115874ef93636a559ab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:26.118382 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6a0a881fcec43b291148f071e4474fc with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:26.118633 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet 981ec6067d624100bddba4383b417c50 (table rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]) successfully deleted
I20240502 14:16:26.123150 32305 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
W20240502 14:16:26.123342 29388 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.123458 32305 ts_tablet_manager.cc:1367] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 1.065s user 0.056s sys 0.003s
W20240502 14:16:26.124003 29385 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0fcde0e7cbe41f1a521e669b798aa99. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.124289 32305 raft_consensus.cc:348] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:26.124536 32305 raft_consensus.cc:729] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:26.124768 32305 consensus_queue.cc:260] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:26.125331 32305 ts_tablet_manager.cc:1398] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:26.130744 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet a230503874a348f4a19647179fe25123 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:48855)
I20240502 14:16:26.130882 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet a230503874a348f4a19647179fe25123 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:48855: session id = ede5e1f11f254824a335e1e43f40b6d1-a230503874a348f4a19647179fe25123
I20240502 14:16:26.133978 29077 tablet_copy_source_session.cc:205] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.123193 29316 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.135636 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a230503874a348f4a19647179fe25123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.136274 29189 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.137492 32588 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.137987 32588 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.141624 32653 tablet_copy_client.cc:308] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:26.142408 29208 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet e66f9bb135bf4007b1864b63a389fac3 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:49699)
I20240502 14:16:26.142602 29208 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet e66f9bb135bf4007b1864b63a389fac3 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:49699: session id = d653f68cb53b4ed79d5e688987ed8d4c-e66f9bb135bf4007b1864b63a389fac3
I20240502 14:16:26.143765 32273 tablet_copy_client.cc:783] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.144307 32273 tablet_copy_client.cc:647] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.146168 32273 tablet_copy_client.cc:523] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.146278 29208 tablet_copy_source_session.cc:205] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.148151 32653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f9bb135bf4007b1864b63a389fac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.151698 32273 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:26.157332 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-96fb0fc5919d42d7bedd770945935b42 received from {username='slave'} at 127.21.58.71:35201
I20240502 14:16:26.157500 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session b7dacf388234446b9a11ce750978ca75-96fb0fc5919d42d7bedd770945935b42 on tablet 96fb0fc5919d42d7bedd770945935b42 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:26.171248 32627 raft_consensus.cc:2229] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:26.171605 32627 raft_consensus.cc:2258] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:26.172991 32627 ts_tablet_manager.cc:1875] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:26.176527 32558 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:26.179986 31595 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.187808 29188 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.198443 28908 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 5 to 6, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.200896 32653 tablet_copy_client.cc:783] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.201155 32653 tablet_copy_client.cc:647] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.206241 32653 tablet_copy_client.cc:523] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.214761 29054 consensus_queue.cc:237] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:26.218019 31391 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.218717 32330 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.221491 32608 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:26.227368 31391 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:26.229766 29316 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.230288 32330 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.231859 29313 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:26.240378 28911 catalog_manager.cc:5484] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c reported cstate change: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:26.245517 29167 tablet_service.cc:1514] Processing DeleteTablet for tablet 093e29d30b2646d68e84b8b9ca86ca9e with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:26.245918 32658 tablet_replica.cc:330] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:26.246160 32658 raft_consensus.cc:2229] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:26.246515 32658 raft_consensus.cc:2258] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:26.247866 32658 ts_tablet_manager.cc:1875] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:26.248325 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 093e29d30b2646d68e84b8b9ca86ca9e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:26.256237 32658 ts_tablet_manager.cc:1888] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:26.256443 32658 log.cc:1196] T 093e29d30b2646d68e84b8b9ca86ca9e P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/093e29d30b2646d68e84b8b9ca86ca9e
I20240502 14:16:26.257308 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet 093e29d30b2646d68e84b8b9ca86ca9e (table rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]) successfully deleted
W20240502 14:16:26.270220 29387 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.270351 29452 consensus_queue.cc:237] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:26.272660 32436 raft_consensus.cc:1050] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:26.273625 32436 consensus_queue.cc:237] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.275112 32534 ts_tablet_manager.cc:902] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:26.281800 29314 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.284386 31107 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a0a881fcec43b291148f071e4474fc with delete_type TABLET_DATA_TOMBSTONED (TS 3bb852e0a21446bbbe824bdefb16036c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46382
I20240502 14:16:26.302749 31126 raft_consensus.cc:1203] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 14:16:26.316226 28901 catalog_manager.cc:4797] TS 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): delete failed for tablet e6a0a881fcec43b291148f071e4474fc because the tablet was not found. No further retry: Not found: Tablet not found: e6a0a881fcec43b291148f071e4474fc
I20240502 14:16:26.317322 32653 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:26.319262 31126 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.319926 32443 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.322958 29313 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.323616 32443 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:16:26.331292 29125 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.340029 32534 tablet_copy_client.cc:308] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:26.349447 29451 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.350005 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 302978e1f6034b3693b3a0abacdc494e from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:56885)
I20240502 14:16:26.350157 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 302978e1f6034b3693b3a0abacdc494e from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:56885: session id = 3bb852e0a21446bbbe824bdefb16036c-302978e1f6034b3693b3a0abacdc494e
I20240502 14:16:26.350060 32443 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.352572 32443 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.355115 29472 tablet_copy_source_session.cc:205] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.357990 32479 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:26.360742 28911 catalog_manager.cc:5484] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:26.368552 29314 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.371904 32627 ts_tablet_manager.cc:1888] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:26.357831 31126 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.372375 32627 log.cc:1196] T a6eb9dee735a4115874ef93636a559ab P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6eb9dee735a4115874ef93636a559ab
I20240502 14:16:26.359028 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302978e1f6034b3693b3a0abacdc494e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.374331 29313 consensus_queue.cc:237] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:26.375553 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet a6eb9dee735a4115874ef93636a559ab (table rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]) successfully deleted
I20240502 14:16:26.380573 29316 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:26.380726 29386 consensus_peers.cc:485] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f047f74165489bacb4f72ab981b447. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.381320 29054 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.382419 32588 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.383024 32588 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.385273 32479 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:26.385620 32479 ts_tablet_manager.cc:1367] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 1.289s user 0.050s sys 0.012s
I20240502 14:16:26.386870 31595 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.391057 29316 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.393253 32273 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:26.394717 32479 raft_consensus.cc:348] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.394997 32479 raft_consensus.cc:729] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:26.395222 32479 consensus_queue.cc:260] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.395807 32479 ts_tablet_manager.cc:1398] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 14:16:26.396176 32542 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:26.396447 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-a0fcde0e7cbe41f1a521e669b798aa99 received from {username='slave'} at 127.21.58.70:56885
I20240502 14:16:26.396581 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-a0fcde0e7cbe41f1a521e669b798aa99 on tablet a0fcde0e7cbe41f1a521e669b798aa99 with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:26.396595 32542 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:26.396870 32542 ts_tablet_manager.cc:1367] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.505s user 0.072s sys 0.019s
I20240502 14:16:26.402880 32703 ts_tablet_manager.cc:902] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:26.403024 32273 tablet_bootstrap.cc:492] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:26.403466 32273 ts_tablet_manager.cc:1367] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.252s user 0.064s sys 0.004s
I20240502 14:16:26.405239 29189 consensus_queue.cc:237] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.405565 32273 raft_consensus.cc:348] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.406031 32273 raft_consensus.cc:729] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:26.408390 29054 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.410461 31126 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.410909 29315 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:16:26.412906 29125 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.414645 32542 raft_consensus.cc:348] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:26.414968 32542 raft_consensus.cc:729] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:26.415207 32542 consensus_queue.cc:260] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:26.415928 32542 ts_tablet_manager.cc:1398] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:16:26.416018 32443 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.416489 32443 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.416560 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-f4dbd9bc62484dd1998080d289db098f received from {username='slave'} at 127.21.58.70:56885
I20240502 14:16:26.416679 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-f4dbd9bc62484dd1998080d289db098f on tablet f4dbd9bc62484dd1998080d289db098f with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:26.418517 28908 catalog_manager.cc:5484] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.422847 32273 consensus_queue.cc:260] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.420431 32436 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.423631 32273 ts_tablet_manager.cc:1398] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:16:26.424295 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-a230503874a348f4a19647179fe25123 received from {username='slave'} at 127.21.58.69:48855
I20240502 14:16:26.424461 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-a230503874a348f4a19647179fe25123 on tablet a230503874a348f4a19647179fe25123 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:26.424768 31126 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.430038 32159 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:26.430549 32159 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:26.430845 32159 ts_tablet_manager.cc:1367] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 1.461s user 0.051s sys 0.005s
I20240502 14:16:26.439157 32534 tablet_copy_client.cc:783] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.439373 32534 tablet_copy_client.cc:647] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.439445 29451 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.443373 32534 tablet_copy_client.cc:523] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.449211 32605 raft_consensus.cc:1050] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:26.458201 28913 catalog_manager.cc:5484] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 6 to 7, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.461272 32605 consensus_queue.cc:237] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:26.463142 32534 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:26.463133 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d94461e22c4ae182afa2a15f3db198 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:26.464582 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44d94461e22c4ae182afa2a15f3db198 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:26.474828 32724 tablet_replica.cc:330] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:26.475248 32724 raft_consensus.cc:2229] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:26.475009 29315 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.476961 32724 raft_consensus.cc:2258] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:26.481196 32724 ts_tablet_manager.cc:1875] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:26.481387 32703 tablet_copy_client.cc:308] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:26.483112 29315 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.483919 32443 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.482192 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet f4dbd9bc62484dd1998080d289db098f from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:60507)
I20240502 14:16:26.484339 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet f4dbd9bc62484dd1998080d289db098f from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:60507: session id = d653f68cb53b4ed79d5e688987ed8d4c-f4dbd9bc62484dd1998080d289db098f
I20240502 14:16:26.486150 31255 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.486765 32443 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.487210 29472 tablet_copy_source_session.cc:205] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.488276 29452 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.488921 32443 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.502573 32703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dbd9bc62484dd1998080d289db098f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.503471 29452 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.516047 32724 ts_tablet_manager.cc:1888] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:16:26.520318 32159 raft_consensus.cc:348] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.525478 32724 log.cc:1196] T 44d94461e22c4ae182afa2a15f3db198 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/44d94461e22c4ae182afa2a15f3db198
I20240502 14:16:26.527622 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet 44d94461e22c4ae182afa2a15f3db198 (table rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]) successfully deleted
I20240502 14:16:26.528638 32159 raft_consensus.cc:729] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:26.531284 32159 consensus_queue.cc:260] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.532012 32159 ts_tablet_manager.cc:1398] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.101s user 0.006s sys 0.000s
I20240502 14:16:26.528844 28913 catalog_manager.cc:5484] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.533157 29452 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:26.535921 32443 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.537132 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-a0fcde0e7cbe41f1a521e669b798aa99 received from {username='slave'} at 127.21.58.72:60507
I20240502 14:16:26.537256 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-a0fcde0e7cbe41f1a521e669b798aa99 on tablet a0fcde0e7cbe41f1a521e669b798aa99 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:26.546343 32703 tablet_copy_client.cc:783] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.549319 29314 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.551169 29054 consensus_queue.cc:237] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.551926 29189 consensus_queue.cc:237] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
W20240502 14:16:26.553030 29386 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302978e1f6034b3693b3a0abacdc494e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.554738 29056 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:16:26.554994 29387 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.555311 32588 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.555393 31255 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.558410 32588 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.563261 29056 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.568930 29315 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.569785 32721 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.572108 29189 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.572142 28910 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.572660 32553 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.572764 29315 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.574110 29189 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.581548 32605 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.582468 28911 catalog_manager.cc:5484] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.587029 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet e89c7a017200449abcd5d9b9c2bcc912 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:26.602743 31257 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.604001 32720 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.617291 29452 raft_consensus.cc:1261] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.617544 29189 raft_consensus.cc:1261] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.617885 32694 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.618527 32687 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.623297 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e89c7a017200449abcd5d9b9c2bcc912 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:26.624161 31255 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.628278 32273 ts_tablet_manager.cc:902] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:26.628862 31126 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.629395 30992 raft_consensus.cc:1203] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:26.634158 32687 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.635994 29185 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.641103 32553 raft_consensus.cc:1050] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:26.641901 32553 consensus_queue.cc:237] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:26.652577 31257 raft_consensus.cc:1203] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:26.655944 32436 raft_consensus.cc:1050] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:26.656790 32436 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:26.660341 29451 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:26.664520 32724 tablet_replica.cc:330] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:26.664762 32724 raft_consensus.cc:2229] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:26.664433 28910 catalog_manager.cc:5484] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:26.665130 32724 raft_consensus.cc:2258] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:26.666364 32724 ts_tablet_manager.cc:1875] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:16:26.677112 28990 consensus_peers.cc:485] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39835ac7b2bb4c0a8c4b43b010b09706. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.679024 29452 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.679836 32330 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.682317 29189 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.683027 32330 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.695420 31255 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.695571 29054 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.696038 32746 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.697008 31126 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.697237 32746 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.697908 29313 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.698305 32746 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.698599 32436 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.700655 31127 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.658900 31391 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.721799 32703 tablet_copy_client.cc:647] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.724099 32703 tablet_copy_client.cc:523] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.728331 29056 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.728864 30992 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.728937 32588 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.729564 32588 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.737315 29189 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.740103 32588 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.747169 32742 raft_consensus.cc:1013] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: attempt to promote peer 3bb852e0a21446bbbe824bdefb16036c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:16:26.747776 32534 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:26.748540 32534 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:26.748404 32605 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.749033 32534 ts_tablet_manager.cc:1367] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.286s user 0.055s sys 0.008s
I20240502 14:16:26.750005 32534 raft_consensus.cc:348] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:26.750258 32534 raft_consensus.cc:729] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:26.750460 32534 consensus_queue.cc:260] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:26.751026 32534 ts_tablet_manager.cc:1398] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:26.751818 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-302978e1f6034b3693b3a0abacdc494e received from {username='slave'} at 127.21.58.70:56885
I20240502 14:16:26.751960 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-302978e1f6034b3693b3a0abacdc494e on tablet 302978e1f6034b3693b3a0abacdc494e with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:26.752197 29314 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.762454 28910 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.771517 32703 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:26.771157 31255 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.781100 32083 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.787873 28911 catalog_manager.cc:5484] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:26.795116 31126 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.795698 32552 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:26.800546 29057 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:26.804924 29452 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.805037 28911 catalog_manager.cc:5484] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:26.807812 29188 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:26.811640 30992 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:26.815205 29451 consensus_queue.cc:237] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:26.818475 32273 tablet_copy_client.cc:308] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:26.819237 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet a0fcde0e7cbe41f1a521e669b798aa99 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:59701)
I20240502 14:16:26.819439 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet a0fcde0e7cbe41f1a521e669b798aa99 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:59701: session id = ede5e1f11f254824a335e1e43f40b6d1-a0fcde0e7cbe41f1a521e669b798aa99
I20240502 14:16:26.822216 29472 tablet_copy_source_session.cc:205] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.823850 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fcde0e7cbe41f1a521e669b798aa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.824216 29054 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.826229 29189 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
W20240502 14:16:26.833979 29255 consensus_peers.cc:485] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 078ffca14316466aa3906d2ff0567c70. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.837710 29186 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.840677 32724 ts_tablet_manager.cc:1888] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:26.852754 29185 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.853575 32552 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.854218 32724 log.cc:1196] T e89c7a017200449abcd5d9b9c2bcc912 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/e89c7a017200449abcd5d9b9c2bcc912
I20240502 14:16:26.857651 30991 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.859458 32553 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 14:16:26.860714 31257 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.861016 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet e89c7a017200449abcd5d9b9c2bcc912 (table rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]) successfully deleted
I20240502 14:16:26.861799 32605 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.862941 31595 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:26.868001 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63690fff28c84670b15ed02ba1356e33 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:26.861315 32273 tablet_copy_client.cc:783] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.868899 32273 tablet_copy_client.cc:647] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.869436 28913 catalog_manager.cc:5484] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:26.871556 32742 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.872876 30992 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:26.875013 29185 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:26.875861 32615 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:26.876318 32615 tablet_bootstrap.cc:492] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:26.876678 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet 63690fff28c84670b15ed02ba1356e33 with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:26.873579 32273 tablet_copy_client.cc:523] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.882334 32615 ts_tablet_manager.cc:1367] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.787s user 0.039s sys 0.013s
I20240502 14:16:26.883193 32615 raft_consensus.cc:348] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.884979 32615 raft_consensus.cc:729] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:26.885242 32615 consensus_queue.cc:260] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.887593 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd4e4241d42d483ab6203606a4d92141 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:26.889006 31258 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.891578 32615 ts_tablet_manager.cc:1398] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:16:26.892259 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-44d94461e22c4ae182afa2a15f3db198 received from {username='slave'} at 127.21.58.72:49699
I20240502 14:16:26.892382 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-44d94461e22c4ae182afa2a15f3db198 on tablet 44d94461e22c4ae182afa2a15f3db198 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:26.898183 28913 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 6 to 7, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.900569 29057 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.905789 31127 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.908087 29314 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.908756 32436 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.910149 29314 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:26.911226 29035 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4e4241d42d483ab6203606a4d92141 with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:26.915357 32605 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.920655 31595 raft_consensus.cc:1050] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:26.921443 32273 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:26.921603 31595 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.923770 32558 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 14:16:26.923969 28990 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fb0fc5919d42d7bedd770945935b42. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.925562 32558 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:26.925868 32558 ts_tablet_manager.cc:1367] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.749s user 0.041s sys 0.010s
I20240502 14:16:26.926550 29313 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.926754 31125 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.926726 32558 raft_consensus.cc:348] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.926965 32558 raft_consensus.cc:729] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:26.927017 305 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.927196 32558 consensus_queue.cc:260] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:26.927361 32552 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.927979 32558 ts_tablet_manager.cc:1398] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:26.929495 317 ts_tablet_manager.cc:902] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:26.930253 29341 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-d67ee220c86f4fc7bc1db8ae08d668d1 received from {username='slave'} at 127.21.58.69:36693
I20240502 14:16:26.930421 29341 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-d67ee220c86f4fc7bc1db8ae08d668d1 on tablet d67ee220c86f4fc7bc1db8ae08d668d1 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:26.932650 29315 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.933315 32588 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:26.934448 29385 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.939908 29315 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.938482 32552 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.947608 31125 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.951167 29452 consensus_queue.cc:237] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:26.954142 317 tablet_copy_client.cc:308] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:26.954794 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 71f047f74165489bacb4f72ab981b447 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:54103)
I20240502 14:16:26.955020 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 71f047f74165489bacb4f72ab981b447 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:54103: session id = b7dacf388234446b9a11ce750978ca75-71f047f74165489bacb4f72ab981b447
I20240502 14:16:26.957891 28908 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 6 to 7, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:26.958530 29472 tablet_copy_source_session.cc:205] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.959336 29188 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.959995 32552 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.961668 317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f047f74165489bacb4f72ab981b447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.961892 29188 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.965785 31391 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:26.966374 32552 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:26.968109 317 tablet_copy_client.cc:783] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.968465 317 tablet_copy_client.cc:647] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.968158 31391 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:26.969832 29057 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.970665 32552 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:26.971204 29387 consensus_peers.cc:485] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 366dbe98fccd42cb8e369876230984fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.971251 317 tablet_copy_client.cc:523] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.976207 29315 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.976804 32552 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.985262 31390 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:26.986017 32436 consensus_queue.cc:1037] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:26.988425 31390 raft_consensus.cc:858] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
W20240502 14:16:26.988591 31390 raft_consensus.cc:1565] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:26.989974 31390 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:26.991779 317 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:26.996963 32720 raft_consensus.cc:1050] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:26.998231 32720 consensus_queue.cc:237] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:27.000695 29316 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.000967 31595 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:27.001354 32746 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.002269 31389 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.007961 28908 catalog_manager.cc:5484] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.011147 31390 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.011951 32742 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.019726 31125 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.020807 32605 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.022397 32436 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.025388 29189 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:27.025337 31125 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.026453 305 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.026928 31255 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:27.027671 305 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.033465 31390 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.034644 28908 catalog_manager.cc:5484] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:27.052956 29316 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:27.032049 31126 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.059619 29315 raft_consensus.cc:2932] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.080202 32552 raft_consensus.cc:1050] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:27.081030 339 ts_tablet_manager.cc:902] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:27.081207 32552 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:27.081996 316 tablet_replica.cc:330] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:27.082223 316 raft_consensus.cc:2229] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:27.082573 316 raft_consensus.cc:2258] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:27.084864 29057 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:27.084846 31124 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:27.100709 32553 raft_consensus.cc:1050] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:27.101879 32553 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:27.101552 305 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
W20240502 14:16:27.106048 29385 consensus_peers.cc:485] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.106902 31125 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.107647 31390 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.107616 31595 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.109753 31595 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.111064 29314 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.111689 31595 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.113768 32608 raft_consensus.cc:1050] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:27.114307 31595 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.114643 32608 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:27.117274 316 ts_tablet_manager.cc:1875] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:16:27.118431 28990 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fb0fc5919d42d7bedd770945935b42. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.119091 29316 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.119593 29185 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.113642 31255 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.127979 29185 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.128674 32588 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.129081 28910 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.130182 29185 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.139936 29188 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.145447 31125 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.118939 31390 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.176350 29314 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.177630 29054 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.178262 32588 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:27.178753 32588 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:27.182471 31125 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.183878 32553 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:16:27.185423 29388 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.186141 32558 ts_tablet_manager.cc:902] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:27.189033 29452 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:27.189877 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:27.192378 31125 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:27.192941 29316 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:27.192996 32588 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:27.194217 32588 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:27.197152 29316 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.198163 32588 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.205226 360 ts_tablet_manager.cc:902] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:27.205235 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0fcde0e7cbe41f1a521e669b798aa99 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:27.205960 31125 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.209982 28908 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.215157 32330 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.095s
I20240502 14:16:27.233767 29188 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:27.240219 32552 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.241153 32742 raft_consensus.cc:1050] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:27.243577 28910 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 4 to 5, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added, now has a pending config: VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65), VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67), VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66), VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71), NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.244589 32742 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:27.248659 29313 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.248708 32588 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:27.245196 29057 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.249706 32436 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.251372 29316 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.252024 32330 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.252297 31258 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.252887 32436 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.255340 32718 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.257400 31125 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.257517 29313 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.260022 29316 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.262566 29188 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.264869 31126 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.266847 31256 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.272464 28908 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.273375 32605 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.278160 31126 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.276357 28910 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.317354 32083 raft_consensus.cc:1050] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:27.322499 32083 consensus_queue.cc:237] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:27.329260 29316 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.330379 32558 tablet_copy_client.cc:308] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:27.340936 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:27.341122 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 39835ac7b2bb4c0a8c4b43b010b09706 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:48855)
I20240502 14:16:27.341794 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 39835ac7b2bb4c0a8c4b43b010b09706 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:48855: session id = ede5e1f11f254824a335e1e43f40b6d1-39835ac7b2bb4c0a8c4b43b010b09706
I20240502 14:16:27.341735 29452 consensus_queue.cc:237] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:27.345230 31391 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:27.345829 29077 tablet_copy_source_session.cc:205] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.346276 355 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:27.356420 31390 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.357667 30972 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 with delete_type TABLET_DATA_TOMBSTONED (TS ede5e1f11f254824a335e1e43f40b6d1 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47126
I20240502 14:16:27.358084 368 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.359516 31257 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.361158 32558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39835ac7b2bb4c0a8c4b43b010b09706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.362824 29189 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.363643 29451 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:27.366284 29057 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:27.367208 32330 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:27.369966 28913 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65), VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67), VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66), VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71), NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69). New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.371192 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:27.372632 32558 tablet_copy_client.cc:783] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.372850 32558 tablet_copy_client.cc:647] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.374274 32552 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:27.373836 31257 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.375577 29315 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:27.382892 29317 raft_consensus.cc:1261] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:27.383222 339 tablet_copy_client.cc:308] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:27.383591 32552 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:27.383832 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 302978e1f6034b3693b3a0abacdc494e from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:54103)
I20240502 14:16:27.383965 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 302978e1f6034b3693b3a0abacdc494e from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:54103: session id = b7dacf388234446b9a11ce750978ca75-302978e1f6034b3693b3a0abacdc494e
I20240502 14:16:27.386874 29472 tablet_copy_source_session.cc:205] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 14:16:27.389330 28899 catalog_manager.cc:4810] TS ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): delete failed for tablet a0fcde0e7cbe41f1a521e669b798aa99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0fcde0e7cbe41f1a521e669b798aa99 already in progress: copying tablet
I20240502 14:16:27.389740 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:27.390290 31126 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:27.391393 32552 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:27.391805 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:27.396713 32552 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.400367 31126 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.407271 29313 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:27.408059 355 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:27.409852 316 ts_tablet_manager.cc:1888] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:27.410038 316 log.cc:1196] T 63690fff28c84670b15ed02ba1356e33 P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/63690fff28c84670b15ed02ba1356e33
I20240502 14:16:27.410914 316 tablet_replica.cc:330] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:27.411320 316 raft_consensus.cc:2229] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:27.411965 29313 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.413846 29188 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:27.414564 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet 63690fff28c84670b15ed02ba1356e33 (table rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]) successfully deleted
I20240502 14:16:27.415452 30992 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:27.416672 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302978e1f6034b3693b3a0abacdc494e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.419837 32558 tablet_copy_client.cc:523] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.420271 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4dbd9bc62484dd1998080d289db098f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:27.423302 32558 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:27.425509 28909 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.428506 29188 raft_consensus.cc:1261] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:27.430344 355 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:27.434671 29388 consensus_peers.cc:485] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3221d1a1b244a27abc9ee26e0a6edb1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.414572 361 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.415343 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:27.421506 29057 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.455992 368 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.445096 29054 consensus_queue.cc:237] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:27.462293 316 raft_consensus.cc:2258] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:27.463963 30992 raft_consensus.cc:1203] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:27.464953 316 ts_tablet_manager.cc:1875] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:27.473453 31370 tablet_service.cc:1514] Processing DeleteTablet for tablet f4dbd9bc62484dd1998080d289db098f with delete_type TABLET_DATA_TOMBSTONED (TS d653f68cb53b4ed79d5e688987ed8d4c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60838
I20240502 14:16:27.476989 29316 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:27.480473 31595 raft_consensus.cc:1050] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:27.481573 31595 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:27.486034 31369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:16:27.486284 29386 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302978e1f6034b3693b3a0abacdc494e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.486593 31126 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.488036 355 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:27.489360 29189 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.490116 355 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:27.493027 345 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:27.495502 29189 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:27.499501 31126 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:27.500274 28910 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.502038 29057 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.502391 361 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.504310 31595 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:27.510984 29057 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:27.518491 368 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:27.519405 29314 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:27.524181 355 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.527194 28913 catalog_manager.cc:5484] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.536005 29189 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.537564 316 ts_tablet_manager.cc:1888] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:16:27.537691 316 log.cc:1196] T fd4e4241d42d483ab6203606a4d92141 P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/fd4e4241d42d483ab6203606a4d92141
I20240502 14:16:27.538468 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet fd4e4241d42d483ab6203606a4d92141 (table rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]) successfully deleted
I20240502 14:16:27.540985 28910 catalog_manager.cc:5484] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.551836 29188 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:27.563736 411 ts_tablet_manager.cc:902] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:27.569041 29057 consensus_queue.cc:237] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:27.572880 30990 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.573192 30991 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:27.577131 368 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.588785 29185 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:27.589473 368 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:27.591523 368 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:27.552121 29317 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.597515 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a230503874a348f4a19647179fe25123 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:27.598686 28913 catalog_manager.cc:5484] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:27.609256 28993 consensus_peers.cc:485] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7bfc3bb327b4407ae1aa7742f4b18de. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.609748 29452 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:27.610419 361 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:27.621625 29188 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:27.624300 339 tablet_copy_client.cc:783] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.624756 339 tablet_copy_client.cc:647] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
W20240502 14:16:27.628484 28902 catalog_manager.cc:4810] TS d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): delete failed for tablet f4dbd9bc62484dd1998080d289db098f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4dbd9bc62484dd1998080d289db098f already in progress: copying tablet
I20240502 14:16:27.629343 420 ts_tablet_manager.cc:902] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:27.633116 339 tablet_copy_client.cc:523] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.636046 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:27.646591 339 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:27.654778 30989 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:27.664088 360 tablet_copy_client.cc:308] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:27.681044 29315 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:27.681680 32083 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:27.689079 32083 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.691676 29451 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.705231 28913 catalog_manager.cc:5484] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:27.732758 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet a230503874a348f4a19647179fe25123 with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:27.741614 29452 consensus_queue.cc:237] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
W20240502 14:16:27.754534 29388 consensus_peers.cc:485] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f047f74165489bacb4f72ab981b447. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:27.754830 29386 consensus_peers.cc:485] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.756240 29054 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:27.756780 31595 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.767290 32653 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.767906 32653 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:27.769454 420 tablet_copy_client.cc:308] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:27.769970 32653 ts_tablet_manager.cc:1367] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 1.452s user 0.058s sys 0.002s
I20240502 14:16:27.770325 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 96fb0fc5919d42d7bedd770945935b42 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:48855)
I20240502 14:16:27.770464 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 96fb0fc5919d42d7bedd770945935b42 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:48855: session id = ede5e1f11f254824a335e1e43f40b6d1-96fb0fc5919d42d7bedd770945935b42
I20240502 14:16:27.771701 32653 raft_consensus.cc:348] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:27.772389 32653 raft_consensus.cc:729] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:27.772555 32653 consensus_queue.cc:260] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:27.773231 411 tablet_copy_client.cc:308] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:27.773373 29077 tablet_copy_source_session.cc:205] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.774768 32653 ts_tablet_manager.cc:1398] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:27.775427 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-e66f9bb135bf4007b1864b63a389fac3 received from {username='slave'} at 127.21.58.72:49699
I20240502 14:16:27.775552 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-e66f9bb135bf4007b1864b63a389fac3 on tablet e66f9bb135bf4007b1864b63a389fac3 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:27.776268 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 366dbe98fccd42cb8e369876230984fe from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:56885)
I20240502 14:16:27.776414 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 366dbe98fccd42cb8e369876230984fe from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:56885: session id = 3bb852e0a21446bbbe824bdefb16036c-366dbe98fccd42cb8e369876230984fe
I20240502 14:16:27.780476 29316 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:27.781056 345 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:27.788774 29451 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:27.790871 29188 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:27.791184 31126 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:27.779029 29472 tablet_copy_source_session.cc:205] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.791560 345 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:27.792043 345 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:27.792183 31390 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:27.795990 345 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:27.797049 32552 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.797610 29188 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.801275 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0fcde0e7cbe41f1a521e669b798aa99 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:27.801489 31126 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.808142 31390 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.811834 28913 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:27.817606 29299 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:27.821609 420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fb0fc5919d42d7bedd770945935b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.822297 411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366dbe98fccd42cb8e369876230984fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.822930 29452 consensus_queue.cc:237] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:27.839359 31595 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.840924 29188 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:27.841470 345 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:27.844056 355 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.846276 29056 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.847785 29185 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.825273 31126 raft_consensus.cc:1261] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:27.852784 345 consensus_queue.cc:1035] T a0fcde0e7cbe41f1a521e669b798aa99 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:27.854612 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet 078ffca14316466aa3906d2ff0567c70 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:55305)
I20240502 14:16:27.854967 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet 078ffca14316466aa3906d2ff0567c70 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:55305: session id = b7dacf388234446b9a11ce750978ca75-078ffca14316466aa3906d2ff0567c70
I20240502 14:16:27.864735 29341 tablet_copy_source_session.cc:205] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.865231 438 tablet_replica.cc:330] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:27.865484 438 raft_consensus.cc:2229] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:27.872366 29314 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.881271 438 raft_consensus.cc:2258] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:27.882272 438 ts_tablet_manager.cc:1875] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:27.882501 360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078ffca14316466aa3906d2ff0567c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.884857 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0fcde0e7cbe41f1a521e669b798aa99 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:27.889096 360 tablet_copy_client.cc:783] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.889561 360 tablet_copy_client.cc:647] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.886145 28909 catalog_manager.cc:5484] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.892074 360 tablet_copy_client.cc:523] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.892432 28913 catalog_manager.cc:5484] T a0fcde0e7cbe41f1a521e669b798aa99 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 10 to 11, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.895154 360 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:27.896006 31370 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 with delete_type TABLET_DATA_TOMBSTONED (TS d653f68cb53b4ed79d5e688987ed8d4c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:60838
I20240502 14:16:27.897431 405 tablet_replica.cc:330] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: stopping tablet replica
I20240502 14:16:27.897912 405 raft_consensus.cc:2229] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:27.898455 405 raft_consensus.cc:2258] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:27.899852 317 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.900239 317 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:27.900470 317 ts_tablet_manager.cc:1367] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.909s user 0.066s sys 0.020s
I20240502 14:16:27.900519 31126 raft_consensus.cc:2932] T a0fcde0e7cbe41f1a521e669b798aa99 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.901269 317 raft_consensus.cc:348] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:27.901494 317 raft_consensus.cc:729] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:27.901705 317 consensus_queue.cc:260] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:27.903007 405 ts_tablet_manager.cc:1875] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:27.903503 29316 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:27.914789 405 ts_tablet_manager.cc:1888] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 14:16:27.916278 405 log.cc:1196] T a0fcde0e7cbe41f1a521e669b798aa99 P d653f68cb53b4ed79d5e688987ed8d4c: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal/wals/a0fcde0e7cbe41f1a521e669b798aa99
I20240502 14:16:27.917353 28902 catalog_manager.cc:4830] TS d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): tablet a0fcde0e7cbe41f1a521e669b798aa99 (table rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]) successfully deleted
I20240502 14:16:27.921797 317 ts_tablet_manager.cc:1398] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 14:16:27.922837 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-71f047f74165489bacb4f72ab981b447 received from {username='slave'} at 127.21.58.71:54103
I20240502 14:16:27.922992 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session b7dacf388234446b9a11ce750978ca75-71f047f74165489bacb4f72ab981b447 on tablet 71f047f74165489bacb4f72ab981b447 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:27.932576 438 ts_tablet_manager.cc:1888] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:27.936009 32653 ts_tablet_manager.cc:902] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:27.944945 29452 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:27.947546 31390 raft_consensus.cc:1203] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:27.949647 31126 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:27.950254 355 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:27.952952 355 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.953714 411 tablet_copy_client.cc:783] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.953924 411 tablet_copy_client.cc:647] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.955901 31126 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.960676 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4dbd9bc62484dd1998080d289db098f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:27.967353 29054 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:27.969637 29185 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:27.970034 28910 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 7 to 8, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:27.970345 361 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:27.971500 411 tablet_copy_client.cc:523] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.972695 31257 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:27.973464 361 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:27.973999 32653 tablet_copy_client.cc:308] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:27.974975 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet d3221d1a1b244a27abc9ee26e0a6edb1 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:60507)
I20240502 14:16:27.975150 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet d3221d1a1b244a27abc9ee26e0a6edb1 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:60507: session id = d653f68cb53b4ed79d5e688987ed8d4c-d3221d1a1b244a27abc9ee26e0a6edb1
I20240502 14:16:27.975827 32703 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.976261 32703 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:27.976506 32703 ts_tablet_manager.cc:1367] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 1.205s user 0.056s sys 0.004s
I20240502 14:16:27.977227 32703 raft_consensus.cc:348] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:27.977474 32703 raft_consensus.cc:729] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:27.977635 32703 consensus_queue.cc:260] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:27.978183 32703 ts_tablet_manager.cc:1398] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:27.981174 368 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.985029 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet f4dbd9bc62484dd1998080d289db098f with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:27.985356 316 tablet_replica.cc:330] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:27.985538 316 raft_consensus.cc:2229] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:27.985953 316 raft_consensus.cc:2258] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:27.986987 316 ts_tablet_manager.cc:1875] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:27.986698 31257 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.990211 29472 tablet_copy_source_session.cc:205] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.991423 32653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3221d1a1b244a27abc9ee26e0a6edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.993499 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96fb0fc5919d42d7bedd770945935b42 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:27.993944 316 ts_tablet_manager.cc:1888] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:16:27.994114 316 log.cc:1196] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/f4dbd9bc62484dd1998080d289db098f
I20240502 14:16:27.995007 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet f4dbd9bc62484dd1998080d289db098f (table rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]) successfully deleted
I20240502 14:16:27.994835 29185 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:27.996191 411 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:28.002333 28910 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.003629 31370 tablet_service.cc:1514] Processing DeleteTablet for tablet f4dbd9bc62484dd1998080d289db098f with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:60838
I20240502 14:16:28.018947 32653 tablet_copy_client.cc:783] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:28.019233 32653 tablet_copy_client.cc:647] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:28.019904 405 tablet_replica.cc:330] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: stopping tablet replica
I20240502 14:16:28.020174 405 raft_consensus.cc:2229] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:16:28.020534 405 raft_consensus.cc:2258] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:16:28.020972 32653 tablet_copy_client.cc:523] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:28.022084 405 ts_tablet_manager.cc:1875] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.022804 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-f4dbd9bc62484dd1998080d289db098f received from {username='slave'} at 127.21.58.72:60507
I20240502 14:16:28.022943 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-f4dbd9bc62484dd1998080d289db098f on tablet f4dbd9bc62484dd1998080d289db098f with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:28.024653 32653 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:28.025960 438 log.cc:1196] T a230503874a348f4a19647179fe25123 P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/a230503874a348f4a19647179fe25123
I20240502 14:16:28.026719 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet a230503874a348f4a19647179fe25123 (table rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]) successfully deleted
I20240502 14:16:27.940857 32694 raft_consensus.cc:1050] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:28.027913 32694 consensus_queue.cc:237] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:28.028993 438 tablet_replica.cc:330] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:28.029196 438 raft_consensus.cc:2229] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:27.957365 29316 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:28.023850 29314 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:28.030396 31595 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:28.030515 368 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:28.032431 29316 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.035321 438 raft_consensus.cc:2258] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:28.041229 29314 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:28.041730 30972 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fb0fc5919d42d7bedd770945935b42 with delete_type TABLET_DATA_TOMBSTONED (TS ede5e1f11f254824a335e1e43f40b6d1 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47126
I20240502 14:16:28.044395 438 ts_tablet_manager.cc:1875] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.051061 32558 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.052331 29452 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.053933 32558 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:28.054767 32273 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.055240 32273 tablet_bootstrap.cc:492] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:28.055476 32558 ts_tablet_manager.cc:1367] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.632s user 0.052s sys 0.003s
I20240502 14:16:28.057730 32558 raft_consensus.cc:348] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
W20240502 14:16:28.058696 29386 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:28.058808 31126 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:28.059484 32552 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:28.061178 32558 raft_consensus.cc:729] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:28.061414 32558 consensus_queue.cc:260] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:28.061987 32558 ts_tablet_manager.cc:1398] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:16:28.059046 420 tablet_copy_client.cc:783] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:28.062032 29054 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:28.062223 420 tablet_copy_client.cc:647] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:28.062525 355 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.064659 420 tablet_copy_client.cc:523] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:28.067385 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 302978e1f6034b3693b3a0abacdc494e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:28.055549 32273 ts_tablet_manager.cc:1367] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 1.169s user 0.048s sys 0.009s
I20240502 14:16:28.068409 31595 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:28.068526 32273 raft_consensus.cc:348] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:28.068818 32273 raft_consensus.cc:729] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:28.068996 32273 consensus_queue.cc:260] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:28.069633 32273 ts_tablet_manager.cc:1398] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:28.070482 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-39835ac7b2bb4c0a8c4b43b010b09706 received from {username='slave'} at 127.21.58.69:48855
I20240502 14:16:28.070631 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-39835ac7b2bb4c0a8c4b43b010b09706 on tablet 39835ac7b2bb4c0a8c4b43b010b09706 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:28.071866 29054 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.076850 30971 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:47126
I20240502 14:16:28.077013 31126 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
W20240502 14:16:28.080118 28899 catalog_manager.cc:4810] TS ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): delete failed for tablet 96fb0fc5919d42d7bedd770945935b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96fb0fc5919d42d7bedd770945935b42 already in progress: copying tablet
W20240502 14:16:28.082000 28899 catalog_manager.cc:4810] TS ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): delete failed for tablet a0fcde0e7cbe41f1a521e669b798aa99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0fcde0e7cbe41f1a521e669b798aa99 already in progress: copying tablet
I20240502 14:16:28.082335 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-a0fcde0e7cbe41f1a521e669b798aa99 received from {username='slave'} at 127.21.58.69:59701
I20240502 14:16:28.082453 420 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:28.082492 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-a0fcde0e7cbe41f1a521e669b798aa99 on tablet a0fcde0e7cbe41f1a521e669b798aa99 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:28.083865 30989 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.087392 28910 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 6 to 7, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:28.090790 29054 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.091794 29452 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.093564 29167 tablet_service.cc:1514] Processing DeleteTablet for tablet 302978e1f6034b3693b3a0abacdc494e with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:28.096592 32703 ts_tablet_manager.cc:902] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:28.106654 30972 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fcde0e7cbe41f1a521e669b798aa99 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:47126
I20240502 14:16:28.107177 471 tablet_replica.cc:330] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: stopping tablet replica
I20240502 14:16:28.107767 471 raft_consensus.cc:2229] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:16:28.108146 471 raft_consensus.cc:2258] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:16:28.109439 471 ts_tablet_manager.cc:1875] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.084331 32694 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.116655 428 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.121785 477 tablet_replica.cc:330] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:28.122056 477 raft_consensus.cc:2229] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:28.121946 478 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.122434 477 raft_consensus.cc:2258] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:28.126029 477 ts_tablet_manager.cc:1875] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.135190 471 ts_tablet_manager.cc:1888] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:16:28.140762 471 log.cc:1196] T a0fcde0e7cbe41f1a521e669b798aa99 P ede5e1f11f254824a335e1e43f40b6d1: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/wals/a0fcde0e7cbe41f1a521e669b798aa99
I20240502 14:16:28.142117 28899 catalog_manager.cc:4830] TS ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): tablet a0fcde0e7cbe41f1a521e669b798aa99 (table rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]) successfully deleted
I20240502 14:16:28.143361 32703 tablet_copy_client.cc:308] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:28.144320 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet b7bfc3bb327b4407ae1aa7742f4b18de from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:39523)
I20240502 14:16:28.144496 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet b7bfc3bb327b4407ae1aa7742f4b18de from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:39523: session id = d653f68cb53b4ed79d5e688987ed8d4c-b7bfc3bb327b4407ae1aa7742f4b18de
I20240502 14:16:28.147648 29077 tablet_copy_source_session.cc:205] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:28.149255 32703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7bfc3bb327b4407ae1aa7742f4b18de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:28.152019 32694 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:28.155867 405 ts_tablet_manager.cc:1888] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:28.156714 30989 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:28.158601 28913 catalog_manager.cc:5484] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:28.162433 339 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.165989 477 ts_tablet_manager.cc:1888] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:16:28.166589 405 log.cc:1196] T f4dbd9bc62484dd1998080d289db098f P d653f68cb53b4ed79d5e688987ed8d4c: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal/wals/f4dbd9bc62484dd1998080d289db098f
I20240502 14:16:28.166874 29056 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:28.169013 28902 catalog_manager.cc:4830] TS d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): tablet f4dbd9bc62484dd1998080d289db098f (table rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]) successfully deleted
I20240502 14:16:28.170024 339 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:28.170322 339 ts_tablet_manager.cc:1367] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.524s user 0.044s sys 0.014s
I20240502 14:16:28.171166 339 raft_consensus.cc:348] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.175206 477 log.cc:1196] T 302978e1f6034b3693b3a0abacdc494e P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/302978e1f6034b3693b3a0abacdc494e
I20240502 14:16:28.176661 339 raft_consensus.cc:729] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:28.177549 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet 302978e1f6034b3693b3a0abacdc494e (table rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]) successfully deleted
I20240502 14:16:28.177875 339 consensus_queue.cc:260] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.184212 29314 consensus_queue.cc:237] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:28.186200 30989 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:28.186415 29056 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:28.186767 428 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:28.187947 428 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:28.189633 32703 tablet_copy_client.cc:783] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:28.189599 428 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:28.190258 32703 tablet_copy_client.cc:647] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:28.179719 339 ts_tablet_manager.cc:1398] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:16:28.192466 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-302978e1f6034b3693b3a0abacdc494e received from {username='slave'} at 127.21.58.71:54103
I20240502 14:16:28.192605 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session b7dacf388234446b9a11ce750978ca75-302978e1f6034b3693b3a0abacdc494e on tablet 302978e1f6034b3693b3a0abacdc494e with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:28.192771 32703 tablet_copy_client.cc:523] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:28.192818 30989 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:28.191121 29056 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:28.199293 28899 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d67ee220c86f4fc7bc1db8ae08d668d1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:28.201902 28910 catalog_manager.cc:5484] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:28.205662 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet d67ee220c86f4fc7bc1db8ae08d668d1 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:28.218739 360 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.220674 360 tablet_bootstrap.cc:492] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:28.222366 32703 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:28.223094 360 ts_tablet_manager.cc:1367] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.328s user 0.050s sys 0.002s
I20240502 14:16:28.226377 360 raft_consensus.cc:348] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.228201 360 raft_consensus.cc:729] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:28.228469 360 consensus_queue.cc:260] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.236760 360 ts_tablet_manager.cc:1398] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 14:16:28.238623 29341 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-078ffca14316466aa3906d2ff0567c70 received from {username='slave'} at 127.21.58.71:55305
I20240502 14:16:28.238776 29341 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session b7dacf388234446b9a11ce750978ca75-078ffca14316466aa3906d2ff0567c70 on tablet 078ffca14316466aa3906d2ff0567c70 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:28.248394 494 tablet_replica.cc:330] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:28.248704 494 raft_consensus.cc:2229] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:28.248975 494 pending_rounds.cc:62] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Trying to abort 1 pending ops.
I20240502 14:16:28.249133 494 pending_rounds.cc:68] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023244853359980544 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d67ee220c86f4fc7bc1db8ae08d668d1" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } } }
I20240502 14:16:28.251010 494 raft_consensus.cc:2894] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 14:16:28.251215 494 raft_consensus.cc:2866] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:16:28.251384 494 raft_consensus.cc:2258] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:28.262673 438 ts_tablet_manager.cc:1888] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:16:28.264962 494 ts_tablet_manager.cc:1875] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.278899 29314 consensus_queue.cc:237] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.279640 32653 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.280223 32653 tablet_bootstrap.cc:492] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:28.280584 32653 ts_tablet_manager.cc:1367] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.256s user 0.047s sys 0.004s
I20240502 14:16:28.281520 438 log.cc:1196] T a0fcde0e7cbe41f1a521e669b798aa99 P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/a0fcde0e7cbe41f1a521e669b798aa99
I20240502 14:16:28.282285 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet a0fcde0e7cbe41f1a521e669b798aa99 (table rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]) successfully deleted
I20240502 14:16:28.281463 32653 raft_consensus.cc:348] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:28.283130 32653 raft_consensus.cc:729] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:28.283411 32653 consensus_queue.cc:260] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:28.287055 32653 ts_tablet_manager.cc:1398] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:16:28.288602 420 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.288682 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-d3221d1a1b244a27abc9ee26e0a6edb1 received from {username='slave'} at 127.21.58.72:60507
I20240502 14:16:28.288841 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-d3221d1a1b244a27abc9ee26e0a6edb1 on tablet d3221d1a1b244a27abc9ee26e0a6edb1 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:28.293879 420 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:28.294974 420 ts_tablet_manager.cc:1367] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.213s user 0.042s sys 0.010s
I20240502 14:16:28.294973 29185 raft_consensus.cc:1261] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:28.295882 420 raft_consensus.cc:348] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:28.297941 478 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:28.303222 420 raft_consensus.cc:729] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:28.303486 420 consensus_queue.cc:260] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:28.305893 420 ts_tablet_manager.cc:1398] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
W20240502 14:16:28.306475 29255 consensus_peers.cc:485] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4818cd7d9b44111a04a0d32a4580cdf. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:28.306829 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-96fb0fc5919d42d7bedd770945935b42 received from {username='slave'} at 127.21.58.69:48855
I20240502 14:16:28.306952 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-96fb0fc5919d42d7bedd770945935b42 on tablet 96fb0fc5919d42d7bedd770945935b42 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:28.307740 30972 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fb0fc5919d42d7bedd770945935b42 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:47126
I20240502 14:16:28.307857 29054 raft_consensus.cc:1261] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:28.308122 471 tablet_replica.cc:330] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: stopping tablet replica
I20240502 14:16:28.308321 471 raft_consensus.cc:2229] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:16:28.308490 471 raft_consensus.cc:2258] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:16:28.308423 478 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:28.310093 471 ts_tablet_manager.cc:1875] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.317039 471 ts_tablet_manager.cc:1888] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:16:28.317209 471 log.cc:1196] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/wals/96fb0fc5919d42d7bedd770945935b42
I20240502 14:16:28.318059 28899 catalog_manager.cc:4830] TS ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): tablet 96fb0fc5919d42d7bedd770945935b42 (table rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]) successfully deleted
I20240502 14:16:28.325310 29185 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:28.330871 32653 ts_tablet_manager.cc:902] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:28.346624 476 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.351411 31255 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:28.351719 29056 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:28.352255 32605 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:28.352526 447 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:28.361070 30989 raft_consensus.cc:1203] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:28.362123 29057 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.368046 31390 raft_consensus.cc:1203] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:28.372905 28910 catalog_manager.cc:5484] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.390763 411 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.399823 411 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:28.400750 411 ts_tablet_manager.cc:1367] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.405s user 0.050s sys 0.006s
I20240502 14:16:28.402089 411 raft_consensus.cc:348] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:28.403589 411 raft_consensus.cc:729] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:28.404031 411 consensus_queue.cc:260] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:28.405128 411 ts_tablet_manager.cc:1398] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:16:28.405921 494 ts_tablet_manager.cc:1888] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:28.406529 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-366dbe98fccd42cb8e369876230984fe received from {username='slave'} at 127.21.58.70:56885
I20240502 14:16:28.406596 494 log.cc:1196] T d67ee220c86f4fc7bc1db8ae08d668d1 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/d67ee220c86f4fc7bc1db8ae08d668d1
I20240502 14:16:28.406684 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-366dbe98fccd42cb8e369876230984fe on tablet 366dbe98fccd42cb8e369876230984fe with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:28.412539 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet d67ee220c86f4fc7bc1db8ae08d668d1 (table rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]) successfully deleted
I20240502 14:16:28.438030 32605 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:28.438735 29185 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.446710 31258 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:28.447187 446 raft_consensus.cc:1050] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:28.448153 446 consensus_queue.cc:237] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:28.466393 31257 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:28.479219 31126 raft_consensus.cc:1203] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:28.498117 28909 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:28.499774 31390 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.500802 29452 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.502645 447 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.503232 447 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.508783 32703 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.509225 32703 tablet_bootstrap.cc:492] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:28.509467 32703 ts_tablet_manager.cc:1367] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.287s user 0.058s sys 0.016s
I20240502 14:16:28.510255 32703 raft_consensus.cc:348] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:28.510499 32703 raft_consensus.cc:729] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:28.510768 32703 consensus_queue.cc:260] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:28.510175 29056 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:28.517724 29054 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.518450 31255 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:28.519373 361 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:28.519650 32703 ts_tablet_manager.cc:1398] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240502 14:16:28.520475 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-b7bfc3bb327b4407ae1aa7742f4b18de received from {username='slave'} at 127.21.58.72:39523
I20240502 14:16:28.520596 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-b7bfc3bb327b4407ae1aa7742f4b18de on tablet b7bfc3bb327b4407ae1aa7742f4b18de with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:28.522675 29314 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:28.523312 361 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:28.525457 368 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.528147 31255 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.529507 29314 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.534688 31391 raft_consensus.cc:1203] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:28.534739 447 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.536985 32653 tablet_copy_client.cc:308] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:28.537815 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 71f047f74165489bacb4f72ab981b447 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:60507)
I20240502 14:16:28.538020 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 71f047f74165489bacb4f72ab981b447 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:60507: session id = d653f68cb53b4ed79d5e688987ed8d4c-71f047f74165489bacb4f72ab981b447
I20240502 14:16:28.541127 28909 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 7 to 8, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.541862 29472 tablet_copy_source_session.cc:205] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:28.542523 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96fb0fc5919d42d7bedd770945935b42 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:28.543771 32653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f047f74165489bacb4f72ab981b447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:28.552145 32653 tablet_copy_client.cc:783] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:28.552604 32653 tablet_copy_client.cc:647] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:28.553154 32605 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.554252 32653 tablet_copy_client.cc:523] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:28.554554 31257 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:28.555228 31595 consensus_queue.cc:1037] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:28.558199 31257 raft_consensus.cc:858] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.558565 32653 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:28.559674 29167 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fb0fc5919d42d7bedd770945935b42 with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:28.560011 477 tablet_replica.cc:330] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:28.560248 477 raft_consensus.cc:2229] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:28.559953 29452 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.560655 477 raft_consensus.cc:2258] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:28.561586 29054 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:28.564236 477 ts_tablet_manager.cc:1875] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:16:28.568253 29121 consensus_peers.cc:485] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 981ec6067d624100bddba4383b417c50. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:28.568667 31390 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.573976 29054 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.577095 28908 catalog_manager.cc:5484] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c reported cstate change: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
W20240502 14:16:28.578431 31257 raft_consensus.cc:1565] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.585016 29185 consensus_queue.cc:237] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:28.585350 477 ts_tablet_manager.cc:1888] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:16:28.587582 31390 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:28.591519 477 log.cc:1196] T 96fb0fc5919d42d7bedd770945935b42 P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/96fb0fc5919d42d7bedd770945935b42
I20240502 14:16:28.588224 32605 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:28.597168 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet 96fb0fc5919d42d7bedd770945935b42 (table rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]) successfully deleted
I20240502 14:16:28.605755 528 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.610193 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e66f9bb135bf4007b1864b63a389fac3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:28.612677 28913 catalog_manager.cc:5484] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:28.612643 31391 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.619963 31257 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.636343 29452 consensus_queue.cc:237] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:28.637725 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet e66f9bb135bf4007b1864b63a389fac3 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:28.638046 494 tablet_replica.cc:330] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:28.638342 494 raft_consensus.cc:2229] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:28.638690 494 raft_consensus.cc:2258] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:28.640112 32653 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:28.640129 494 ts_tablet_manager.cc:1875] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.640563 32653 tablet_bootstrap.cc:492] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:28.640838 32653 ts_tablet_manager.cc:1367] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.082s user 0.053s sys 0.011s
I20240502 14:16:28.641644 32653 raft_consensus.cc:348] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:28.641937 32653 raft_consensus.cc:729] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:28.643007 355 raft_consensus.cc:1050] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:28.643748 355 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:28.644271 31255 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:28.650058 31126 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:28.650864 29056 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:28.642160 32653 consensus_queue.cc:260] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:28.651571 32653 ts_tablet_manager.cc:1398] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20240502 14:16:28.651968 32552 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:28.652566 32552 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:28.653661 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-71f047f74165489bacb4f72ab981b447 received from {username='slave'} at 127.21.58.72:60507
I20240502 14:16:28.653793 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-71f047f74165489bacb4f72ab981b447 on tablet 71f047f74165489bacb4f72ab981b447 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:28.662472 31595 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.668033 29056 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.679394 28910 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.684456 494 ts_tablet_manager.cc:1888] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:28.685376 494 log.cc:1196] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/e66f9bb135bf4007b1864b63a389fac3
I20240502 14:16:28.687397 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet e66f9bb135bf4007b1864b63a389fac3 (table rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]) successfully deleted
I20240502 14:16:28.690836 31126 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.694893 31257 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:28.698261 31257 raft_consensus.cc:1203] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:28.700209 355 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:28.702185 31257 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:28.705160 29057 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:28.705986 32605 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:28.707736 29057 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.720530 29185 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:28.721223 355 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:28.722252 29057 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:28.723415 355 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:28.725494 29387 consensus_peers.cc:485] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d95b84899c994d45a569d5a4dda124ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:28.728400 355 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:28.729089 29185 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:28.739303 29057 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:28.740173 28915 catalog_manager.cc:5484] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index -1 to 4, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:28.764362 31390 raft_consensus.cc:1203] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:16:28.804558 420 ts_tablet_manager.cc:902] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:28.854815 32552 raft_consensus.cc:1050] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:28.855710 32552 consensus_queue.cc:237] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:28.862694 29057 consensus_queue.cc:237] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:28.878098 29185 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:28.879933 28991 consensus_peers.cc:485] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 239ee601d297412494cf2b447ef75e9a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:28.880903 29314 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:28.881561 32330 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:28.888698 361 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:28.926663 607 ts_tablet_manager.cc:902] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:28.930828 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:28.931617 29316 raft_consensus.cc:1261] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.931663 31390 raft_consensus.cc:1261] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.932489 355 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.933111 355 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:28.936259 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:28.957947 564 raft_consensus.cc:1050] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:28.959475 31595 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.961509 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:28.963388 564 consensus_queue.cc:237] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:28.964659 32330 raft_consensus.cc:1050] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:28.965457 32330 consensus_queue.cc:237] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:28.967062 31391 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.968773 29316 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.973309 28908 catalog_manager.cc:5484] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c reported cstate change: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.979004 29452 consensus_queue.cc:237] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:28.981678 31390 raft_consensus.cc:1261] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:28.981853 29316 raft_consensus.cc:1261] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:28.982705 31595 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:28.985370 31595 consensus_queue.cc:1035] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:28.988392 355 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:28.992514 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3221d1a1b244a27abc9ee26e0a6edb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:28.994056 29185 raft_consensus.cc:1261] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:28.993777 29316 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.001243 32552 raft_consensus.cc:1050] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:29.002947 32552 consensus_queue.cc:237] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:29.016345 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:29.019506 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:29.030531 31126 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.031273 355 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.032136 28912 catalog_manager.cc:5484] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:29.035324 29452 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.035933 368 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.036783 29167 tablet_service.cc:1514] Processing DeleteTablet for tablet d3221d1a1b244a27abc9ee26e0a6edb1 with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:29.037250 477 tablet_replica.cc:330] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:29.037531 477 raft_consensus.cc:2229] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.037801 477 raft_consensus.cc:2258] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.038753 477 ts_tablet_manager.cc:1875] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.044071 361 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:29.054504 477 ts_tablet_manager.cc:1888] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:29.054235 29316 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:29.058773 477 log.cc:1196] T d3221d1a1b244a27abc9ee26e0a6edb1 P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3221d1a1b244a27abc9ee26e0a6edb1
I20240502 14:16:29.060788 30989 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.061436 368 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.065703 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet d3221d1a1b244a27abc9ee26e0a6edb1 (table rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]) successfully deleted
I20240502 14:16:29.069034 28908 catalog_manager.cc:5484] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:29.069890 368 raft_consensus.cc:1050] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:29.070667 368 consensus_queue.cc:237] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:29.074877 29057 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.081794 564 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:16:29.103492 29388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.66:44753, user_credentials={real_user=slave}} blocked reactor thread for 107749us
I20240502 14:16:29.106573 31390 raft_consensus.cc:2932] T d3221d1a1b244a27abc9ee26e0a6edb1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.115612 29314 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.116441 640 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.118309 29452 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.118971 640 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.120657 31390 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.121227 640 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.122380 640 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:29.123963 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:29.127338 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:29.126783 30989 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
W20240502 14:16:29.128816 29385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.67:40039, user_credentials={real_user=slave}} blocked reactor thread for 63892.8us
I20240502 14:16:29.129683 29315 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.131657 355 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.134830 29057 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.137370 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:29.141571 31126 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.148716 28908 catalog_manager.cc:5484] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.156160 29451 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:29.160781 31124 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:29.161223 607 tablet_copy_client.cc:308] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:29.161280 31595 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:29.161916 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet c4818cd7d9b44111a04a0d32a4580cdf from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:55305)
I20240502 14:16:29.162067 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet c4818cd7d9b44111a04a0d32a4580cdf from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:55305: session id = b7dacf388234446b9a11ce750978ca75-c4818cd7d9b44111a04a0d32a4580cdf
I20240502 14:16:29.162892 31255 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:29.164371 31595 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:29.164695 31124 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.164891 31595 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.165264 29341 tablet_copy_source_session.cc:205] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:29.167534 607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4818cd7d9b44111a04a0d32a4580cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:29.167510 29188 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.169775 29185 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:29.170992 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:29.173593 31255 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.175560 368 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.176493 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 302978e1f6034b3693b3a0abacdc494e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:29.176954 660 ts_tablet_manager.cc:902] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:29.178740 31255 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.180749 564 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.181985 31391 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.183295 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:29.199982 630 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.201901 643 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.209365 29185 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:29.215488 29054 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.215736 28913 catalog_manager.cc:5484] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:29.217936 32552 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.226305 29167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:29.226408 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:29.178695 28908 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.230985 29054 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.231742 355 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.237291 29316 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.240381 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet 302978e1f6034b3693b3a0abacdc494e with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:29.242780 665 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.250696 32330 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.252624 29313 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.254498 29316 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.261946 28912 catalog_manager.cc:5484] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
W20240502 14:16:29.262528 29386 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.71:44703, user_credentials={real_user=slave}} blocked reactor thread for 78598.6us
I20240502 14:16:29.274746 28913 catalog_manager.cc:5484] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.275568 420 tablet_copy_client.cc:308] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:29.288682 29451 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.315690 29449 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:29.315469 31390 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.324090 29315 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.324549 29035 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:29.326674 29056 consensus_queue.cc:237] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:29.335592 607 tablet_copy_client.cc:783] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
W20240502 14:16:29.341637 29121 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.67:40039, user_credentials={real_user=slave}} blocked reactor thread for 51766.4us
I20240502 14:16:29.342589 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:29.342511 31391 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.345203 672 tablet_replica.cc:330] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:29.345484 672 raft_consensus.cc:2229] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.345974 672 raft_consensus.cc:2258] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.346123 29057 consensus_queue.cc:237] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:29.346524 30990 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.347826 32330 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.349224 31389 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.353607 640 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.354889 660 tablet_copy_client.cc:308] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:29.356125 651 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.358263 29185 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.359194 607 tablet_copy_client.cc:647] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:29.360841 361 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.364529 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7bfc3bb327b4407ae1aa7742f4b18de with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:29.366581 31389 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.371994 643 raft_consensus.cc:1050] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:29.372359 28908 catalog_manager.cc:5484] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:29.373394 643 consensus_queue.cc:237] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:29.376498 29208 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet 981ec6067d624100bddba4383b417c50 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:47337)
I20240502 14:16:29.376657 29208 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet 981ec6067d624100bddba4383b417c50 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:47337: session id = ede5e1f11f254824a335e1e43f40b6d1-981ec6067d624100bddba4383b417c50
I20240502 14:16:29.380051 29208 tablet_copy_source_session.cc:205] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:29.381723 420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981ec6067d624100bddba4383b417c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:29.387821 31389 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:29.390573 420 tablet_copy_client.cc:783] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:29.392573 32330 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:29.391055 420 tablet_copy_client.cc:647] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:29.394093 643 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:29.396708 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39835ac7b2bb4c0a8c4b43b010b09706 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:29.397138 420 tablet_copy_client.cc:523] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:29.402390 31257 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.406754 688 ts_tablet_manager.cc:902] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:29.408608 420 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:29.408491 29185 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:29.409855 29054 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:29.411850 31255 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:29.413528 28912 catalog_manager.cc:5484] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:29.414402 666 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:29.416893 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet d95b84899c994d45a569d5a4dda124ed from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:56885)
I20240502 14:16:29.417055 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet d95b84899c994d45a569d5a4dda124ed from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:56885: session id = 3bb852e0a21446bbbe824bdefb16036c-d95b84899c994d45a569d5a4dda124ed
I20240502 14:16:29.417171 29451 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.418208 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet 39835ac7b2bb4c0a8c4b43b010b09706 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:29.419951 29472 tablet_copy_source_session.cc:205] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:29.425164 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bfc3bb327b4407ae1aa7742f4b18de with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:29.425923 686 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.429618 692 tablet_replica.cc:330] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:29.431361 607 tablet_copy_client.cc:523] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:29.431740 355 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240502 14:16:29.432098 692 raft_consensus.cc:2229] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.432502 692 raft_consensus.cc:2258] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.433423 29451 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.437242 690 tablet_replica.cc:330] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:29.438894 690 raft_consensus.cc:2229] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.439946 692 ts_tablet_manager.cc:1875] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.440136 607 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:29.440404 690 raft_consensus.cc:2258] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.441951 690 ts_tablet_manager.cc:1875] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.448864 660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95b84899c994d45a569d5a4dda124ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:29.460525 665 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.461014 29057 consensus_queue.cc:237] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:29.463930 692 ts_tablet_manager.cc:1888] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:29.464061 692 log.cc:1196] T b7bfc3bb327b4407ae1aa7742f4b18de P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7bfc3bb327b4407ae1aa7742f4b18de
I20240502 14:16:29.464032 29054 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.467701 690 ts_tablet_manager.cc:1888] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:29.467869 690 log.cc:1196] T 39835ac7b2bb4c0a8c4b43b010b09706 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/39835ac7b2bb4c0a8c4b43b010b09706
I20240502 14:16:29.470105 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet 39835ac7b2bb4c0a8c4b43b010b09706 (table rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]) successfully deleted
I20240502 14:16:29.471194 31391 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.479241 31255 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.480329 672 ts_tablet_manager.cc:1875] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.485335 28909 catalog_manager.cc:5484] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 6 to 7, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.502090 29449 consensus_queue.cc:237] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:29.504441 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:29.506094 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:29.506669 31391 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:29.507366 663 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:29.507704 29054 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:29.508810 643 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:29.511343 665 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.514020 29056 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.518114 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71f047f74165489bacb4f72ab981b447 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:29.518981 31391 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.520921 31257 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:29.521602 32552 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:29.523964 28909 catalog_manager.cc:5484] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 7 to 8, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:29.526041 29316 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:29.526810 672 ts_tablet_manager.cc:1888] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:16:29.526943 672 log.cc:1196] T 302978e1f6034b3693b3a0abacdc494e P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/302978e1f6034b3693b3a0abacdc494e
I20240502 14:16:29.528362 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet 302978e1f6034b3693b3a0abacdc494e (table rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]) successfully deleted
I20240502 14:16:29.537125 30990 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:29.537496 31255 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:29.582433 688 tablet_copy_client.cc:308] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:29.585068 660 tablet_copy_client.cc:783] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:29.585462 660 tablet_copy_client.cc:647] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:29.589114 660 tablet_copy_client.cc:523] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:16:29.599159 29254 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.69:36535, user_credentials={real_user=slave}} blocked reactor thread for 163740us
I20240502 14:16:29.599525 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet b7bfc3bb327b4407ae1aa7742f4b18de (table rebalance_test_table_7 [id=c623a6a1d9954161b2d78dcce7d02b20]) successfully deleted
I20240502 14:16:29.599939 693 raft_consensus.cc:1050] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:29.600797 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f047f74165489bacb4f72ab981b447 with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:29.601490 692 tablet_replica.cc:330] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:29.601711 692 raft_consensus.cc:2229] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.602041 692 raft_consensus.cc:2258] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.606495 692 ts_tablet_manager.cc:1875] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.611786 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 239ee601d297412494cf2b447ef75e9a from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:35201)
I20240502 14:16:29.611994 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 239ee601d297412494cf2b447ef75e9a from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:35201: session id = b7dacf388234446b9a11ce750978ca75-239ee601d297412494cf2b447ef75e9a
I20240502 14:16:29.601013 693 consensus_queue.cc:237] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:29.621642 692 ts_tablet_manager.cc:1888] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:16:29.622229 692 log.cc:1196] T 71f047f74165489bacb4f72ab981b447 P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/71f047f74165489bacb4f72ab981b447
I20240502 14:16:29.624421 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet 71f047f74165489bacb4f72ab981b447 (table rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]) successfully deleted
I20240502 14:16:29.627562 660 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:29.631956 31255 raft_consensus.cc:1261] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.637152 29077 tablet_copy_source_session.cc:205] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:29.643548 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee601d297412494cf2b447ef75e9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:16:29.649387 28993 consensus_peers.cc:485] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0272c80ea41149f895074e0e16180b22. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:29.649945 29451 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:29.650652 361 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:29.666649 688 tablet_copy_client.cc:783] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:29.666890 688 tablet_copy_client.cc:647] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:29.672500 32330 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.674824 29185 raft_consensus.cc:1261] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.678773 29449 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.686985 694 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
W20240502 14:16:29.687629 29256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.65:46783, user_credentials={real_user=slave}} blocked reactor thread for 49694.5us
I20240502 14:16:29.688284 29449 raft_consensus.cc:1261] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:29.689631 28912 catalog_manager.cc:5484] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.693011 708 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:29.698483 688 tablet_copy_client.cc:523] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:29.701867 688 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:29.715821 714 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:16:29.720278 29255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.71:44703, user_credentials={real_user=slave}} blocked reactor thread for 82456.4us
I20240502 14:16:29.728960 29316 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:29.730793 714 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.741595 29449 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.748458 28909 catalog_manager.cc:5484] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.757388 29449 consensus_queue.cc:237] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:29.759601 31126 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.760180 355 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.760186 29185 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.765553 630 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.766288 31126 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.769680 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 366dbe98fccd42cb8e369876230984fe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:29.772617 28909 catalog_manager.cc:5484] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:29.774312 29316 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.774907 630 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.776422 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet 366dbe98fccd42cb8e369876230984fe with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:29.776799 672 tablet_replica.cc:330] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:29.777004 672 raft_consensus.cc:2229] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.777235 672 raft_consensus.cc:2258] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.778195 672 ts_tablet_manager.cc:1875] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.779577 29316 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:29.781968 361 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:29.790218 672 ts_tablet_manager.cc:1888] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:29.790402 672 log.cc:1196] T 366dbe98fccd42cb8e369876230984fe P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/366dbe98fccd42cb8e369876230984fe
I20240502 14:16:29.793488 31255 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.796380 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet 366dbe98fccd42cb8e369876230984fe (table rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]) successfully deleted
I20240502 14:16:29.806221 29315 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:29.844920 420 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:29.852541 607 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:29.853307 607 tablet_bootstrap.cc:492] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:29.854010 420 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:29.854519 607 ts_tablet_manager.cc:1367] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.416s user 0.038s sys 0.013s
I20240502 14:16:29.856099 607 raft_consensus.cc:348] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:29.856647 420 ts_tablet_manager.cc:1367] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.448s user 0.051s sys 0.016s
I20240502 14:16:29.862573 420 raft_consensus.cc:348] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:29.862959 607 raft_consensus.cc:729] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:29.863274 607 consensus_queue.cc:260] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:29.863608 420 raft_consensus.cc:729] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:29.863904 420 consensus_queue.cc:260] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:29.864568 607 ts_tablet_manager.cc:1398] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:16:29.865165 420 ts_tablet_manager.cc:1398] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:16:29.866225 29341 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-c4818cd7d9b44111a04a0d32a4580cdf received from {username='slave'} at 127.21.58.71:55305
I20240502 14:16:29.866390 29341 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session b7dacf388234446b9a11ce750978ca75-c4818cd7d9b44111a04a0d32a4580cdf on tablet c4818cd7d9b44111a04a0d32a4580cdf with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:29.868026 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-981ec6067d624100bddba4383b417c50 received from {username='slave'} at 127.21.58.69:47337
I20240502 14:16:29.868176 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-981ec6067d624100bddba4383b417c50 on tablet 981ec6067d624100bddba4383b417c50 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:29.874665 30990 raft_consensus.cc:1203] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:29.876142 29316 consensus_queue.cc:237] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:29.883095 31255 raft_consensus.cc:1261] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.884192 29185 raft_consensus.cc:1261] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:29.884833 693 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.885385 693 consensus_queue.cc:1035] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:29.890376 660 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:29.890640 29185 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.891613 31255 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.897122 660 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:29.897789 28915 catalog_manager.cc:5484] T 078ffca14316466aa3906d2ff0567c70 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.898658 660 ts_tablet_manager.cc:1367] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.271s user 0.048s sys 0.002s
I20240502 14:16:29.899497 660 raft_consensus.cc:348] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:29.899788 660 raft_consensus.cc:729] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:29.900019 660 consensus_queue.cc:260] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:29.901572 660 ts_tablet_manager.cc:1398] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:29.902257 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-d95b84899c994d45a569d5a4dda124ed received from {username='slave'} at 127.21.58.70:56885
I20240502 14:16:29.902410 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-d95b84899c994d45a569d5a4dda124ed on tablet d95b84899c994d45a569d5a4dda124ed with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:29.907155 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet 078ffca14316466aa3906d2ff0567c70 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:29.894857 693 raft_consensus.cc:2932] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:29.907753 690 tablet_replica.cc:330] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:29.907982 690 raft_consensus.cc:2229] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.908366 690 raft_consensus.cc:2258] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.913832 28899 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 078ffca14316466aa3906d2ff0567c70 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:29.914106 690 ts_tablet_manager.cc:1875] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.935158 690 ts_tablet_manager.cc:1888] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:29.939704 690 log.cc:1196] T 078ffca14316466aa3906d2ff0567c70 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/078ffca14316466aa3906d2ff0567c70
I20240502 14:16:29.940882 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet 078ffca14316466aa3906d2ff0567c70 (table rebalance_test_table_4 [id=dd071eb2b4d148b7b60d6e9293fbcdd8]) successfully deleted
I20240502 14:16:29.999677 29185 consensus_queue.cc:237] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:30.007905 31255 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.008527 32605 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.014878 29316 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.015619 32605 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:30.018174 29124 consensus_peers.cc:485] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e89c7a017200449abcd5d9b9c2bcc912. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:30.022135 32605 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:30.026552 29316 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:30.028370 28908 catalog_manager.cc:5484] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:30.041337 31255 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:30.078049 29449 consensus_queue.cc:237] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:30.080312 31255 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:30.081071 355 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:30.081225 31391 raft_consensus.cc:1261] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:30.082237 666 consensus_queue.cc:1035] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:30.084720 643 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:30.085912 31391 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:30.086505 31257 raft_consensus.cc:2932] T 71f047f74165489bacb4f72ab981b447 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:30.091399 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71f047f74165489bacb4f72ab981b447 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:30.096128 28913 catalog_manager.cc:5484] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 8 to 9, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:30.101011 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f047f74165489bacb4f72ab981b447 with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:30.101421 672 tablet_replica.cc:330] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:30.101620 672 raft_consensus.cc:2229] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:30.102017 672 raft_consensus.cc:2258] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:30.103185 672 ts_tablet_manager.cc:1875] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:30.110824 672 ts_tablet_manager.cc:1888] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:16:30.110985 672 log.cc:1196] T 71f047f74165489bacb4f72ab981b447 P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/71f047f74165489bacb4f72ab981b447
I20240502 14:16:30.112068 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet 71f047f74165489bacb4f72ab981b447 (table rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]) successfully deleted
I20240502 14:16:30.114991 768 ts_tablet_manager.cc:902] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:30.132282 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:30.150939 31255 raft_consensus.cc:1203] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:30.163604 29316 consensus_queue.cc:237] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
W20240502 14:16:30.179610 29254 consensus_peers.cc:485] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a00d9c46bfc64a4085715e4f764b8c96. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:30.180696 29449 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:30.181322 713 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:30.190119 29057 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:30.190706 713 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:30.200585 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:30.204850 768 tablet_copy_client.cc:308] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:30.205615 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 0272c80ea41149f895074e0e16180b22 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:39523)
I20240502 14:16:30.205806 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 0272c80ea41149f895074e0e16180b22 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:39523: session id = d653f68cb53b4ed79d5e688987ed8d4c-0272c80ea41149f895074e0e16180b22
I20240502 14:16:30.209332 29077 tablet_copy_source_session.cc:205] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:30.210986 768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0272c80ea41149f895074e0e16180b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:30.223448 768 tablet_copy_client.cc:783] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:30.224035 768 tablet_copy_client.cc:647] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:30.226886 768 tablet_copy_client.cc:523] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:30.232138 713 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:30.235867 29056 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:30.236828 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.240516 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.244148 28909 catalog_manager.cc:5484] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:30.246174 31124 raft_consensus.cc:1203] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:30.250041 768 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:30.266553 753 raft_consensus.cc:1050] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:30.267799 753 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:30.269119 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:30.278204 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.302333 29449 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:30.308612 31255 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:30.309970 681 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:30.314824 29057 consensus_queue.cc:237] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:30.321890 29056 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:30.322674 681 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:30.328650 688 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:30.338642 688 tablet_bootstrap.cc:492] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:30.338786 31390 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.339401 686 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.341015 29451 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:16:30.341913 28990 consensus_peers.cc:485] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fde0741532a4067a4806f4bc0b0d465. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:30.342396 688 ts_tablet_manager.cc:1367] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.641s user 0.050s sys 0.009s
I20240502 14:16:30.343326 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:30.343374 688 raft_consensus.cc:348] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:30.343621 688 raft_consensus.cc:729] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:30.343667 686 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.343825 688 consensus_queue.cc:260] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:30.344390 688 ts_tablet_manager.cc:1398] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:30.355911 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-239ee601d297412494cf2b447ef75e9a received from {username='slave'} at 127.21.58.71:35201
I20240502 14:16:30.356070 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session b7dacf388234446b9a11ce750978ca75-239ee601d297412494cf2b447ef75e9a on tablet 239ee601d297412494cf2b447ef75e9a with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:30.372664 30990 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:30.373777 753 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 14:16:30.374876 681 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:30.380789 31257 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:30.392324 686 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:30.396031 31391 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:30.400305 28910 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:30.403713 28909 catalog_manager.cc:5484] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:30.411757 29056 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:30.418823 29451 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:30.430891 31257 raft_consensus.cc:1203] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:30.434414 30990 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:30.451539 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.457402 29057 consensus_queue.cc:237] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:30.495093 31390 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.495725 361 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:30.503147 28992 consensus_peers.cc:485] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 093e29d30b2646d68e84b8b9ca86ca9e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:30.504209 29315 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.504778 361 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.515017 29185 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:30.526718 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.531477 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:30.531380 361 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:30.535404 29314 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:30.537904 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:30.538303 681 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:30.538614 31257 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:30.539664 31124 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:30.540203 681 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:30.543644 681 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:30.543356 29315 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:30.552910 693 raft_consensus.cc:1050] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:30.554044 693 consensus_queue.cc:237] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:30.558099 28910 catalog_manager.cc:5484] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:30.580075 29056 raft_consensus.cc:1261] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.581506 31255 raft_consensus.cc:1261] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.582018 713 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.583935 713 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.591156 29185 raft_consensus.cc:1261] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.591924 713 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.616560 32605 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.617301 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.623615 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:30.628175 31258 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.633767 29315 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.639811 31391 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:30.641857 31124 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.642156 28908 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.648020 846 ts_tablet_manager.cc:902] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:30.648738 29316 consensus_queue.cc:237] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:30.672140 713 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.676132 29188 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.677424 29189 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:30.680863 31124 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:30.681392 32605 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:30.681953 31257 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:30.682560 32605 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:30.685271 32605 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.689303 31125 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.695384 28908 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.697016 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd4e4241d42d483ab6203606a4d92141 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:30.701388 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4e4241d42d483ab6203606a4d92141 with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:30.701479 31258 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.701504 31255 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.710525 28913 catalog_manager.cc:5484] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.714223 768 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:30.723930 29185 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
W20240502 14:16:30.727881 29124 consensus_peers.cc:485] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e -> Peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): Couldn't send request to peer 139e612d62744c02bcb5653b7aa1dc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd4e4241d42d483ab6203606a4d92141. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:30.732103 31257 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:30.732689 32605 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:30.734459 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.735785 753 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:30.740397 28902 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fd4e4241d42d483ab6203606a4d92141 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:16:30.740918 31257 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:30.745236 768 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:30.745555 768 ts_tablet_manager.cc:1367] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.496s user 0.052s sys 0.010s
I20240502 14:16:30.746460 768 raft_consensus.cc:348] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:30.746970 31124 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:30.747169 768 raft_consensus.cc:729] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:30.747579 681 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:30.747996 768 consensus_queue.cc:260] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:30.750074 31126 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:30.750953 29056 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.751817 768 ts_tablet_manager.cc:1398] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:16:30.755415 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-0272c80ea41149f895074e0e16180b22 received from {username='slave'} at 127.21.58.72:39523
I20240502 14:16:30.755545 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-0272c80ea41149f895074e0e16180b22 on tablet 0272c80ea41149f895074e0e16180b22 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:30.756969 28908 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 10 to 11, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:30.764048 30989 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.765372 713 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.768704 29315 consensus_queue.cc:237] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:30.768705 861 tablet_replica.cc:330] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:30.769639 861 raft_consensus.cc:2229] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:30.770032 861 raft_consensus.cc:2258] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:30.771464 29057 raft_consensus.cc:1261] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:30.772002 694 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:30.781786 694 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.783638 861 ts_tablet_manager.cc:1875] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:30.787652 28899 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4818cd7d9b44111a04a0d32a4580cdf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:30.788808 29057 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.795615 865 ts_tablet_manager.cc:902] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:30.795670 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:30.804253 28913 catalog_manager.cc:5484] T c4818cd7d9b44111a04a0d32a4580cdf P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.806466 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:30.813107 29315 consensus_queue.cc:237] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:30.815557 29316 consensus_queue.cc:237] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:30.820880 28910 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet c4818cd7d9b44111a04a0d32a4580cdf with cas_config_opid_index 5: aborting the task: latest config opid_index 6; task opid_index 5
I20240502 14:16:30.821353 31257 raft_consensus.cc:1261] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:30.822197 841 consensus_queue.cc:1035] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:30.824113 31257 raft_consensus.cc:2932] T c4818cd7d9b44111a04a0d32a4580cdf P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.828347 29168 tablet_service.cc:1514] Processing DeleteTablet for tablet c4818cd7d9b44111a04a0d32a4580cdf with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:30.832039 841 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:30.836565 874 tablet_replica.cc:330] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:30.838577 874 raft_consensus.cc:2229] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:30.839463 874 raft_consensus.cc:2258] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:30.841051 30989 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:16:30.841481 29255 consensus_peers.cc:485] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 432dc9a199294b748414520a2c49a674. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:30.841858 694 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.842391 30989 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.843416 29451 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.843819 713 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.844374 713 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:30.850037 29257 consensus_peers.cc:485] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d67ee220c86f4fc7bc1db8ae08d668d1. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:30.850256 29257 consensus_peers.cc:485] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f17889f58a34bc0afa40a246e5debc6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:30.850422 686 raft_consensus.cc:1050] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:30.851692 686 consensus_queue.cc:237] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:30.866096 30992 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:30.871165 694 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:30.872414 28910 catalog_manager.cc:5484] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:30.874527 29056 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.842360 29185 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:30.879808 693 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.880398 874 ts_tablet_manager.cc:1875] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:30.883267 643 raft_consensus.cc:1050] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:30.886381 693 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:30.889336 29188 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.889513 30991 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:30.890053 32330 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.890695 846 tablet_copy_client.cc:308] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:30.891384 29208 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet e89c7a017200449abcd5d9b9c2bcc912 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:45939)
I20240502 14:16:30.891539 29208 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet e89c7a017200449abcd5d9b9c2bcc912 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:45939: session id = 3bb852e0a21446bbbe824bdefb16036c-e89c7a017200449abcd5d9b9c2bcc912
I20240502 14:16:30.895666 29208 tablet_copy_source_session.cc:205] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:30.897120 846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89c7a017200449abcd5d9b9c2bcc912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:30.899931 31257 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.901332 28913 catalog_manager.cc:5484] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:30.903009 686 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.904515 846 tablet_copy_client.cc:783] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:30.904904 846 tablet_copy_client.cc:647] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:30.910651 846 tablet_copy_client.cc:523] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:30.911579 643 consensus_queue.cc:237] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:30.916040 29316 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.916661 361 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.918004 846 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
W20240502 14:16:30.926517 29256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.68:46509, user_credentials={real_user=slave}} blocked reactor thread for 82617.2us
I20240502 14:16:30.933766 713 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:30.935958 29185 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:30.941519 861 ts_tablet_manager.cc:1888] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:16:30.941648 861 log.cc:1196] T fd4e4241d42d483ab6203606a4d92141 P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd4e4241d42d483ab6203606a4d92141
I20240502 14:16:30.948585 871 ts_tablet_manager.cc:902] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:30.942476 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet fd4e4241d42d483ab6203606a4d92141 (table rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]) successfully deleted
I20240502 14:16:30.948899 30991 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:30.958000 865 tablet_copy_client.cc:308] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:30.958833 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet a00d9c46bfc64a4085715e4f764b8c96 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:36693)
I20240502 14:16:30.959060 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet a00d9c46bfc64a4085715e4f764b8c96 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:36693: session id = ede5e1f11f254824a335e1e43f40b6d1-a00d9c46bfc64a4085715e4f764b8c96
I20240502 14:16:30.962399 29341 tablet_copy_source_session.cc:205] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:30.966784 31126 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.967556 865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00d9c46bfc64a4085715e4f764b8c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:30.968484 355 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.969399 28910 catalog_manager.cc:5484] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:30.974380 865 tablet_copy_client.cc:783] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:30.974668 865 tablet_copy_client.cc:647] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:30.977809 361 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.979004 865 tablet_copy_client.cc:523] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:30.982709 865 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:30.984745 29315 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:30.985991 28910 catalog_manager.cc:5484] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:30.995307 29185 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.996647 29057 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:30.996795 355 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.997375 355 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:30.993167 29449 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:30.989567 29188 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:31.010502 31257 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:31.049257 874 ts_tablet_manager.cc:1888] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:31.049489 874 log.cc:1196] T c4818cd7d9b44111a04a0d32a4580cdf P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/c4818cd7d9b44111a04a0d32a4580cdf
I20240502 14:16:31.050510 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet c4818cd7d9b44111a04a0d32a4580cdf (table rebalance_test_table_8 [id=a61f61a448354326a1834e5e87d7d2b5]) successfully deleted
I20240502 14:16:31.053818 906 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:31.052122 29057 consensus_queue.cc:237] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:31.061914 29185 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:31.068940 28913 catalog_manager.cc:5484] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:31.071888 29189 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.073175 32330 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:31.077291 29189 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:31.077834 909 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:31.084538 28910 catalog_manager.cc:5484] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:31.089315 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet 239ee601d297412494cf2b447ef75e9a with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:31.095559 29449 consensus_queue.cc:237] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:31.096598 861 tablet_replica.cc:330] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:31.096807 861 raft_consensus.cc:2229] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:31.096647 29057 consensus_queue.cc:237] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:31.098315 861 raft_consensus.cc:2258] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:31.094115 29056 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:31.100503 29189 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.101207 643 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:31.102301 861 ts_tablet_manager.cc:1875] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:31.111322 355 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:31.115348 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d95b84899c994d45a569d5a4dda124ed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:31.119699 28910 catalog_manager.cc:5484] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:31.121413 29185 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:31.124076 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet d95b84899c994d45a569d5a4dda124ed with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:31.128799 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 239ee601d297412494cf2b447ef75e9a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 14:16:31.129596 28990 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.69:36535, user_credentials={real_user=slave}} blocked reactor thread for 84541.6us
I20240502 14:16:31.134044 31390 raft_consensus.cc:1203] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:31.145778 31124 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.151652 875 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:31.154004 871 tablet_copy_client.cc:308] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:31.154105 31257 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.154974 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 3fde0741532a4067a4806f4bc0b0d465 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:48855)
W20240502 14:16:31.155113 28991 consensus_peers.cc:485] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3901ce6537fd48ba861fba13767d8aa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:31.155200 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 3fde0741532a4067a4806f4bc0b0d465 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:48855: session id = ede5e1f11f254824a335e1e43f40b6d1-3fde0741532a4067a4806f4bc0b0d465
I20240502 14:16:31.155788 909 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:31.157594 31257 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:31.158870 29077 tablet_copy_source_session.cc:205] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:31.160380 871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fde0741532a4067a4806f4bc0b0d465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:31.165992 29188 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:31.167888 32330 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:31.173743 871 tablet_copy_client.cc:783] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:31.173978 871 tablet_copy_client.cc:647] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:31.174991 31124 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:31.176017 865 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:31.176051 32330 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:31.176298 871 tablet_copy_client.cc:523] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:31.176488 865 tablet_bootstrap.cc:492] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:31.176836 865 ts_tablet_manager.cc:1367] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.194s user 0.053s sys 0.045s
I20240502 14:16:31.177683 865 raft_consensus.cc:348] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:31.178012 865 raft_consensus.cc:729] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:31.178256 865 consensus_queue.cc:260] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:31.180027 871 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:31.183166 865 ts_tablet_manager.cc:1398] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:16:31.178618 31126 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:31.188984 29341 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-a00d9c46bfc64a4085715e4f764b8c96 received from {username='slave'} at 127.21.58.69:36693
I20240502 14:16:31.189113 29341 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-a00d9c46bfc64a4085715e4f764b8c96 on tablet a00d9c46bfc64a4085715e4f764b8c96 with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:31.194406 32330 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:31.207665 28913 catalog_manager.cc:5484] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:31.211935 29189 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:31.217634 31127 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:31.243640 861 ts_tablet_manager.cc:1888] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:31.243953 861 log.cc:1196] T 239ee601d297412494cf2b447ef75e9a P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/239ee601d297412494cf2b447ef75e9a
I20240502 14:16:31.245009 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet 239ee601d297412494cf2b447ef75e9a (table rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]) successfully deleted
I20240502 14:16:31.268007 926 ts_tablet_manager.cc:902] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:31.272163 921 tablet_replica.cc:330] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:31.272388 921 raft_consensus.cc:2229] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:31.272712 921 pending_rounds.cc:62] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Trying to abort 1 pending ops.
I20240502 14:16:31.272846 921 pending_rounds.cc:68] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023244865059311616 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d95b84899c994d45a569d5a4dda124ed" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } } }
I20240502 14:16:31.273391 921 raft_consensus.cc:2894] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 14:16:31.273523 921 raft_consensus.cc:2866] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:16:31.273640 921 raft_consensus.cc:2258] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:31.275831 921 ts_tablet_manager.cc:1875] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:31.276075 31124 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:31.286427 921 ts_tablet_manager.cc:1888] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:31.286581 921 log.cc:1196] T d95b84899c994d45a569d5a4dda124ed P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/d95b84899c994d45a569d5a4dda124ed
I20240502 14:16:31.288661 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet d95b84899c994d45a569d5a4dda124ed (table rebalance_test_table_5 [id=5d3d62d9036a4b349caec86fc6dcf0da]) successfully deleted
I20240502 14:16:31.314355 30991 raft_consensus.cc:1203] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:31.327548 936 ts_tablet_manager.cc:902] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:31.329823 926 tablet_copy_client.cc:308] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:31.330644 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 093e29d30b2646d68e84b8b9ca86ca9e from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:60009)
I20240502 14:16:31.330816 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 093e29d30b2646d68e84b8b9ca86ca9e from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:60009: session id = 3bb852e0a21446bbbe824bdefb16036c-093e29d30b2646d68e84b8b9ca86ca9e
I20240502 14:16:31.333612 29077 tablet_copy_source_session.cc:205] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:31.336035 926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e29d30b2646d68e84b8b9ca86ca9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:31.342495 943 ts_tablet_manager.cc:902] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:31.347617 846 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:31.348124 939 ts_tablet_manager.cc:902] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:31.357216 926 tablet_copy_client.cc:783] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:31.357529 926 tablet_copy_client.cc:647] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:31.363040 926 tablet_copy_client.cc:523] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:31.367938 949 ts_tablet_manager.cc:902] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
I20240502 14:16:31.369798 846 tablet_bootstrap.cc:492] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:31.370486 846 ts_tablet_manager.cc:1367] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.453s user 0.057s sys 0.002s
I20240502 14:16:31.371377 846 raft_consensus.cc:348] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:31.371661 846 raft_consensus.cc:729] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:31.371883 846 consensus_queue.cc:260] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:31.372579 846 ts_tablet_manager.cc:1398] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:31.374650 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-e89c7a017200449abcd5d9b9c2bcc912 received from {username='slave'} at 127.21.58.70:45939
I20240502 14:16:31.374780 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-e89c7a017200449abcd5d9b9c2bcc912 on tablet e89c7a017200449abcd5d9b9c2bcc912 with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:31.380110 939 tablet_copy_client.cc:308] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:31.379815 943 tablet_copy_client.cc:308] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:31.381445 29208 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet fd4e4241d42d483ab6203606a4d92141 from peer 139e612d62744c02bcb5653b7aa1dc29 ({username='slave'} at 127.21.58.68:45613)
I20240502 14:16:31.381597 29208 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet fd4e4241d42d483ab6203606a4d92141 from peer 139e612d62744c02bcb5653b7aa1dc29 at {username='slave'} at 127.21.58.68:45613: session id = 139e612d62744c02bcb5653b7aa1dc29-fd4e4241d42d483ab6203606a4d92141
I20240502 14:16:31.381930 926 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:31.386646 29208 tablet_copy_source_session.cc:205] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:31.388531 943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4e4241d42d483ab6203606a4d92141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:31.388507 936 tablet_copy_client.cc:308] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:31.389209 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet 432dc9a199294b748414520a2c49a674 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:55305)
I20240502 14:16:31.389338 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet 432dc9a199294b748414520a2c49a674 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:55305: session id = b7dacf388234446b9a11ce750978ca75-432dc9a199294b748414520a2c49a674
I20240502 14:16:31.392473 29341 tablet_copy_source_session.cc:205] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:31.399107 943 tablet_copy_client.cc:783] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:31.399462 943 tablet_copy_client.cc:647] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:31.401681 936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432dc9a199294b748414520a2c49a674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:31.401711 943 tablet_copy_client.cc:523] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:31.415558 943 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:31.443543 936 tablet_copy_client.cc:783] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:31.443858 936 tablet_copy_client.cc:647] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:31.448904 936 tablet_copy_client.cc:523] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:31.452709 936 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:31.458150 949 tablet_copy_client.cc:308] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40039
I20240502 14:16:31.488073 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet 1f17889f58a34bc0afa40a246e5debc6 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:44461)
I20240502 14:16:31.488291 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet 1f17889f58a34bc0afa40a246e5debc6 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:44461: session id = d653f68cb53b4ed79d5e688987ed8d4c-1f17889f58a34bc0afa40a246e5debc6
I20240502 14:16:31.491221 29341 tablet_copy_source_session.cc:205] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:31.492604 939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f17889f58a34bc0afa40a246e5debc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:31.495277 29341 tablet_copy_service.cc:140] P 1b3c20f7bfd548769f6bd40148590b74: Received BeginTabletCopySession request for tablet d67ee220c86f4fc7bc1db8ae08d668d1 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:44461)
I20240502 14:16:31.495473 29341 tablet_copy_service.cc:161] P 1b3c20f7bfd548769f6bd40148590b74: Beginning new tablet copy session on tablet d67ee220c86f4fc7bc1db8ae08d668d1 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:44461: session id = d653f68cb53b4ed79d5e688987ed8d4c-d67ee220c86f4fc7bc1db8ae08d668d1
I20240502 14:16:31.498332 939 tablet_copy_client.cc:783] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:31.498479 29341 tablet_copy_source_session.cc:205] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:31.498586 939 tablet_copy_client.cc:647] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:31.501048 949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67ee220c86f4fc7bc1db8ae08d668d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:31.505398 939 tablet_copy_client.cc:523] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:31.510350 31124 raft_consensus.cc:1203] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:31.518003 939 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:31.529878 949 tablet_copy_client.cc:783] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:31.530164 949 tablet_copy_client.cc:647] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:31.535830 29449 consensus_queue.cc:237] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:31.538802 29185 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:31.539476 355 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:31.539645 31124 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:31.539783 871 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:31.540977 643 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:31.541396 871 tablet_bootstrap.cc:492] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:31.541739 871 ts_tablet_manager.cc:1367] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.362s user 0.070s sys 0.026s
I20240502 14:16:31.541827 949 tablet_copy_client.cc:523] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:31.541863 643 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:31.542757 871 raft_consensus.cc:348] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:31.543183 871 raft_consensus.cc:729] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:31.543452 871 consensus_queue.cc:260] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:31.544205 871 ts_tablet_manager.cc:1398] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:31.544584 31124 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:31.544932 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-3fde0741532a4067a4806f4bc0b0d465 received from {username='slave'} at 127.21.58.69:48855
I20240502 14:16:31.545078 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-3fde0741532a4067a4806f4bc0b0d465 on tablet 3fde0741532a4067a4806f4bc0b0d465 with peer ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:31.541864 29386 consensus_peers.cc:485] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d95b84899c994d45a569d5a4dda124ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:31.546733 29185 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:31.557169 28913 catalog_manager.cc:5484] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:31.586882 949 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:31.590070 936 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:31.590494 936 tablet_bootstrap.cc:492] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:31.590759 936 ts_tablet_manager.cc:1367] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.138s user 0.045s sys 0.013s
I20240502 14:16:31.591487 936 raft_consensus.cc:348] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:31.591766 936 raft_consensus.cc:729] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:31.591979 936 consensus_queue.cc:260] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:31.592492 936 ts_tablet_manager.cc:1398] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:16:31.599726 29340 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-432dc9a199294b748414520a2c49a674 received from {username='slave'} at 127.21.58.71:55305
I20240502 14:16:31.599895 29340 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session b7dacf388234446b9a11ce750978ca75-432dc9a199294b748414520a2c49a674 on tablet 432dc9a199294b748414520a2c49a674 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:31.624200 943 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:31.624742 943 tablet_bootstrap.cc:492] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap complete.
I20240502 14:16:31.625059 943 ts_tablet_manager.cc:1367] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.210s user 0.048s sys 0.007s
I20240502 14:16:31.625916 943 raft_consensus.cc:348] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:31.626202 943 raft_consensus.cc:729] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: LEARNER
I20240502 14:16:31.626417 943 consensus_queue.cc:260] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:31.637125 943 ts_tablet_manager.cc:1398] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 14:16:31.641134 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session 139e612d62744c02bcb5653b7aa1dc29-fd4e4241d42d483ab6203606a4d92141 received from {username='slave'} at 127.21.58.68:45613
I20240502 14:16:31.641302 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session 139e612d62744c02bcb5653b7aa1dc29-fd4e4241d42d483ab6203606a4d92141 on tablet fd4e4241d42d483ab6203606a4d92141 with peer 139e612d62744c02bcb5653b7aa1dc29
I20240502 14:16:31.662482 32330 raft_consensus.cc:1050] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:31.663633 32330 consensus_queue.cc:237] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:31.674871 31390 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:31.675539 686 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:31.684099 29449 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:31.684330 29316 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:31.684794 686 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:31.685111 914 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:31.688550 29054 consensus_queue.cc:237] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:31.714521 30990 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:31.716168 29185 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:31.716446 939 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 14:16:31.716871 28992 consensus_peers.cc:485] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39835ac7b2bb4c0a8c4b43b010b09706. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:31.716925 939 tablet_bootstrap.cc:492] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:31.717186 939 ts_tablet_manager.cc:1367] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.199s user 0.049s sys 0.006s
I20240502 14:16:31.717726 686 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:31.718286 686 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:31.721755 926 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:31.721469 914 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.722324 926 tablet_bootstrap.cc:492] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:31.722656 926 ts_tablet_manager.cc:1367] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.341s user 0.067s sys 0.030s
I20240502 14:16:31.724251 926 raft_consensus.cc:348] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:31.724479 926 raft_consensus.cc:729] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:31.724694 926 consensus_queue.cc:260] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:31.725317 926 ts_tablet_manager.cc:1398] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:31.727180 939 raft_consensus.cc:348] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:31.727514 939 raft_consensus.cc:729] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:31.727699 939 consensus_queue.cc:260] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:31.729761 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-093e29d30b2646d68e84b8b9ca86ca9e received from {username='slave'} at 127.21.58.70:60009
I20240502 14:16:31.729936 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-093e29d30b2646d68e84b8b9ca86ca9e on tablet 093e29d30b2646d68e84b8b9ca86ca9e with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:31.730170 31391 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.731994 693 raft_consensus.cc:1050] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:31.732826 693 consensus_queue.cc:237] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:31.740198 29316 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.743563 29449 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.758064 949 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:31.759781 939 ts_tablet_manager.cc:1398] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.042s user 0.005s sys 0.001s
I20240502 14:16:31.760964 29340 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-1f17889f58a34bc0afa40a246e5debc6 received from {username='slave'} at 127.21.58.72:44461
I20240502 14:16:31.761096 29340 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-1f17889f58a34bc0afa40a246e5debc6 on tablet 1f17889f58a34bc0afa40a246e5debc6 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:31.763228 28913 catalog_manager.cc:5484] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 4 to 5, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.764106 949 tablet_bootstrap.cc:492] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap complete.
I20240502 14:16:31.767554 949 ts_tablet_manager.cc:1367] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent bootstrapping tablet: real 0.181s user 0.044s sys 0.015s
I20240502 14:16:31.768791 949 raft_consensus.cc:348] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:31.769256 949 raft_consensus.cc:729] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d653f68cb53b4ed79d5e688987ed8d4c, State: Initialized, Role: LEARNER
I20240502 14:16:31.769593 949 consensus_queue.cc:260] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:31.770628 949 ts_tablet_manager.cc:1398] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:16:31.773159 29340 tablet_copy_service.cc:342] P 1b3c20f7bfd548769f6bd40148590b74: Request end of tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-d67ee220c86f4fc7bc1db8ae08d668d1 received from {username='slave'} at 127.21.58.72:44461
I20240502 14:16:31.773284 29340 tablet_copy_service.cc:433] P 1b3c20f7bfd548769f6bd40148590b74: ending tablet copy session d653f68cb53b4ed79d5e688987ed8d4c-d67ee220c86f4fc7bc1db8ae08d668d1 on tablet d67ee220c86f4fc7bc1db8ae08d668d1 with peer d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:31.775415 30990 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:31.776086 841 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:31.780843 29054 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:31.781489 841 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:31.789464 29449 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:31.790163 841 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:31.802505 29185 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:31.811564 32330 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:31.818763 31391 raft_consensus.cc:1203] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:31.828202 29451 raft_consensus.cc:1203] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 14:16:31.864558 31391 raft_consensus.cc:1203] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:31.866286 936 ts_tablet_manager.cc:902] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:31.868810 841 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.872128 29057 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.874378 30990 raft_consensus.cc:1203] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:31.874219 30989 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.878613 31257 raft_consensus.cc:1203] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:31.879653 29451 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.881994 28910 catalog_manager.cc:5484] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 4 to 5, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.888070 28913 catalog_manager.cc:5484] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:31.893213 29057 consensus_queue.cc:237] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:31.894635 29316 consensus_queue.cc:237] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:31.896696 31391 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.897413 1023 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:31.898232 30991 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.899963 889 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:31.900262 1023 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.900652 29057 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.901837 30989 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:31.902472 841 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.903738 31391 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.907622 29316 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:31.908330 1025 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:31.911590 693 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:31.913755 936 tablet_copy_client.cc:308] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:31.913436 29057 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.913619 28913 catalog_manager.cc:5484] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:31.914690 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 3901ce6537fd48ba861fba13767d8aa4 from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:35201)
I20240502 14:16:31.914839 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 3901ce6537fd48ba861fba13767d8aa4 from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:35201: session id = b7dacf388234446b9a11ce750978ca75-3901ce6537fd48ba861fba13767d8aa4
I20240502 14:16:31.915045 28899 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a00d9c46bfc64a4085715e4f764b8c96 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:31.917899 29077 tablet_copy_source_session.cc:205] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:31.918536 29316 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.919363 936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901ce6537fd48ba861fba13767d8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:31.921077 30989 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:31.927008 28913 catalog_manager.cc:5484] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 5 to 6, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:31.930334 936 tablet_copy_client.cc:783] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:31.930588 936 tablet_copy_client.cc:647] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:31.934659 936 tablet_copy_client.cc:523] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:31.940600 936 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:31.942272 753 raft_consensus.cc:1050] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:31.943056 753 consensus_queue.cc:237] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:31.968261 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet a00d9c46bfc64a4085715e4f764b8c96 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:31.968571 29430 tablet_service.cc:1514] Processing DeleteTablet for tablet 0272c80ea41149f895074e0e16180b22 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:31.968606 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0272c80ea41149f895074e0e16180b22 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:31.981032 31124 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:31.981431 1047 tablet_replica.cc:330] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:31.981742 1047 raft_consensus.cc:2229] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:31.981735 681 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:31.982152 1047 raft_consensus.cc:2258] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:31.983419 1047 ts_tablet_manager.cc:1875] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:31.986043 31257 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:31.986639 681 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:31.992475 1047 ts_tablet_manager.cc:1888] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:31.994640 29316 raft_consensus.cc:1261] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:31.995406 681 consensus_queue.cc:1035] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:31.998241 1047 log.cc:1196] T a00d9c46bfc64a4085715e4f764b8c96 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/a00d9c46bfc64a4085715e4f764b8c96
I20240502 14:16:31.999233 1047 tablet_replica.cc:330] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:32.000753 1047 raft_consensus.cc:2229] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:32.001112 1047 raft_consensus.cc:2258] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:32.003055 1047 ts_tablet_manager.cc:1875] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:32.014616 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet a00d9c46bfc64a4085715e4f764b8c96 (table rebalance_test_table_3 [id=b097cc0423da401bbcf8fbbdc9ccd9aa]) successfully deleted
I20240502 14:16:32.018240 1047 ts_tablet_manager.cc:1888] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:16:32.020397 1047 log.cc:1196] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/0272c80ea41149f895074e0e16180b22
I20240502 14:16:32.025425 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet 0272c80ea41149f895074e0e16180b22 (table rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]) successfully deleted
I20240502 14:16:32.028056 681 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.035488 31124 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.036546 31255 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.037736 936 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:32.038451 936 tablet_bootstrap.cc:492] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:32.038708 936 ts_tablet_manager.cc:1367] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.098s user 0.046s sys 0.018s
I20240502 14:16:32.039708 936 raft_consensus.cc:348] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:32.039961 936 raft_consensus.cc:729] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:32.040251 936 consensus_queue.cc:260] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:32.040845 28915 catalog_manager.cc:5484] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:32.051853 29316 raft_consensus.cc:2932] T e89c7a017200449abcd5d9b9c2bcc912 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.070406 31125 raft_consensus.cc:1203] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:32.072230 1067 ts_tablet_manager.cc:902] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:32.116751 29449 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:32.119604 29316 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:32.119789 31257 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:32.120433 355 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:32.120994 355 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:16:32.120908 29385 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6a0a881fcec43b291148f071e4474fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:32.129294 1067 tablet_copy_client.cc:308] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:32.129976 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet d95b84899c994d45a569d5a4dda124ed from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:54103)
I20240502 14:16:32.130141 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet d95b84899c994d45a569d5a4dda124ed from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:54103: session id = b7dacf388234446b9a11ce750978ca75-d95b84899c994d45a569d5a4dda124ed
I20240502 14:16:32.134553 29472 tablet_copy_source_session.cc:205] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:32.135437 643 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:32.136117 1067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95b84899c994d45a569d5a4dda124ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:32.141505 28915 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:32.141309 31255 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:32.146027 29315 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:32.170557 936 ts_tablet_manager.cc:1398] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.132s user 0.020s sys 0.056s
I20240502 14:16:32.172582 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-3901ce6537fd48ba861fba13767d8aa4 received from {username='slave'} at 127.21.58.71:35201
I20240502 14:16:32.172825 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session b7dacf388234446b9a11ce750978ca75-3901ce6537fd48ba861fba13767d8aa4 on tablet 3901ce6537fd48ba861fba13767d8aa4 with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:32.225494 1067 tablet_copy_client.cc:783] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:32.225865 1067 tablet_copy_client.cc:647] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:32.230417 1067 tablet_copy_client.cc:523] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:32.277168 1067 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:32.289577 693 raft_consensus.cc:1050] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:32.290673 693 consensus_queue.cc:237] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:32.315246 30989 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.316020 841 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.317185 31391 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.317780 841 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.337172 29057 raft_consensus.cc:1261] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.343034 693 consensus_queue.cc:1035] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.350868 841 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.356132 30989 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.358264 29057 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.361523 1121 ts_tablet_manager.cc:902] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:32.369779 28910 catalog_manager.cc:5484] T d67ee220c86f4fc7bc1db8ae08d668d1 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.379341 31391 raft_consensus.cc:2932] T d67ee220c86f4fc7bc1db8ae08d668d1 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.436968 32605 raft_consensus.cc:1050] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 to VOTER
I20240502 14:16:32.437949 32605 consensus_queue.cc:237] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } }
I20240502 14:16:32.438951 29185 consensus_queue.cc:237] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:32.444679 1023 raft_consensus.cc:1050] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:32.445744 1023 consensus_queue.cc:237] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:32.456365 693 raft_consensus.cc:1050] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c to VOTER
I20240502 14:16:32.457340 693 consensus_queue.cc:237] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:32.475570 31391 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:32.476147 31390 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.476373 681 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:32.477017 1121 tablet_copy_client.cc:308] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:32.477701 841 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.477769 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 39835ac7b2bb4c0a8c4b43b010b09706 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:60009)
I20240502 14:16:32.477948 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 39835ac7b2bb4c0a8c4b43b010b09706 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:60009: session id = 3bb852e0a21446bbbe824bdefb16036c-39835ac7b2bb4c0a8c4b43b010b09706
I20240502 14:16:32.478789 30989 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:16:32.479028 29124 consensus_peers.cc:485] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e -> Peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): Couldn't send request to peer 139e612d62744c02bcb5653b7aa1dc29. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:32.479329 1130 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.480901 29077 tablet_copy_source_session.cc:205] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:32.488258 1121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39835ac7b2bb4c0a8c4b43b010b09706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:32.499699 29185 raft_consensus.cc:1261] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.500602 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:32.500531 693 consensus_queue.cc:1035] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.506716 1131 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.507627 31391 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.511881 30989 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.528005 28915 catalog_manager.cc:5484] T 1f17889f58a34bc0afa40a246e5debc6 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.534792 29185 raft_consensus.cc:2932] T 1f17889f58a34bc0afa40a246e5debc6 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } }
I20240502 14:16:32.592073 31255 raft_consensus.cc:1203] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:32.591872 1063 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:32.607797 28915 catalog_manager.cc:5484] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:32.617337 29449 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:32.618304 681 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:32.619374 1121 tablet_copy_client.cc:783] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:32.619693 1121 tablet_copy_client.cc:647] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:32.627614 30989 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.628444 1159 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.641019 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:32.642015 29451 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.642556 31255 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:32.642629 1025 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.644656 681 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:32.655723 1121 tablet_copy_client.cc:523] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:32.656508 31125 raft_consensus.cc:1261] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:32.657084 681 consensus_queue.cc:1035] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:32.669492 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:32.681829 29057 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:32.691254 32605 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:32.695427 1121 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
W20240502 14:16:32.695936 28993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.72:36549, user_credentials={real_user=slave}} blocked reactor thread for 224966us
I20240502 14:16:32.696106 31390 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:32.697525 31389 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.702275 1022 raft_consensus.cc:1050] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:32.702952 1022 consensus_queue.cc:237] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:32.709224 31125 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.709846 1155 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.723345 31388 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.724273 1023 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.724366 29315 raft_consensus.cc:1261] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.726346 889 raft_consensus.cc:1050] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:32.727327 889 consensus_queue.cc:237] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:32.729537 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:32.736472 1025 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.737596 1025 consensus_queue.cc:1035] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 14:16:32.751893 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:32.761018 1194 ts_tablet_manager.cc:902] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:32.783775 1063 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:32.785072 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:32.786793 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:32.786540 1025 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.794279 31257 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:32.795552 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:32.802758 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:32.800846 1155 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.809826 29451 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.809728 30989 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.817667 28909 catalog_manager.cc:5484] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.820446 31124 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.821223 31257 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.821805 693 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.831120 30990 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.831668 693 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.833315 29057 consensus_queue.cc:237] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:32.835752 28915 catalog_manager.cc:5484] T fd4e4241d42d483ab6203606a4d92141 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 11 to 12, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:32.837312 31389 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.842098 29449 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:32.843307 31388 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.849691 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:32.861394 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:32.861222 28909 catalog_manager.cc:5484] T 093e29d30b2646d68e84b8b9ca86ca9e P d653f68cb53b4ed79d5e688987ed8d4c reported cstate change: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.862718 31391 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:32.863415 1153 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:32.870878 1067 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:32.871441 1067 tablet_bootstrap.cc:492] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:32.871742 1067 ts_tablet_manager.cc:1367] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.595s user 0.065s sys 0.003s
I20240502 14:16:32.872491 1067 raft_consensus.cc:348] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:32.872749 1067 raft_consensus.cc:729] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:32.872939 1067 consensus_queue.cc:260] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:32.872879 29057 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
W20240502 14:16:32.873445 29123 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.71:44703, user_credentials={real_user=slave}} blocked reactor thread for 73722.9us
I20240502 14:16:32.876631 29451 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:32.877249 693 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:32.811388 31125 raft_consensus.cc:2932] T fd4e4241d42d483ab6203606a4d92141 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:32.891134 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:32.891505 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:32.893748 29315 raft_consensus.cc:2932] T 093e29d30b2646d68e84b8b9ca86ca9e P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:32.913437 1153 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.914770 31388 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.916513 1194 tablet_copy_client.cc:308] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:32.917209 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet e6a0a881fcec43b291148f071e4474fc from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:59701)
I20240502 14:16:32.917356 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet e6a0a881fcec43b291148f071e4474fc from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:59701: session id = ede5e1f11f254824a335e1e43f40b6d1-e6a0a881fcec43b291148f071e4474fc
I20240502 14:16:32.925843 1067 ts_tablet_manager.cc:1398] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20240502 14:16:32.926541 29471 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-d95b84899c994d45a569d5a4dda124ed received from {username='slave'} at 127.21.58.71:54103
I20240502 14:16:32.926684 29471 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session b7dacf388234446b9a11ce750978ca75-d95b84899c994d45a569d5a4dda124ed on tablet d95b84899c994d45a569d5a4dda124ed with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:32.933333 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fde0741532a4067a4806f4bc0b0d465 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:32.945012 29472 tablet_copy_source_session.cc:205] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:32.946571 1194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a0a881fcec43b291148f071e4474fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:32.950040 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:32.952121 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:32.958668 29451 consensus_queue.cc:237] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:32.974313 693 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:32.979820 1194 tablet_copy_client.cc:783] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:32.982549 30990 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:32.984082 28915 catalog_manager.cc:5484] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:32.987398 30992 raft_consensus.cc:1261] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:32.988147 1155 consensus_queue.cc:1035] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:32.990792 1194 tablet_copy_client.cc:647] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:32.992115 30992 raft_consensus.cc:2932] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:32.992830 1194 tablet_copy_client.cc:523] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:32.998145 31257 raft_consensus.cc:1203] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:33.001096 28908 catalog_manager.cc:5484] T 3fde0741532a4067a4806f4bc0b0d465 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:33.002228 31257 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.005309 1194 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:33.005722 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fde0741532a4067a4806f4bc0b0d465 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:33.007223 1229 ts_tablet_manager.cc:902] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:33.022680 29452 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.034785 1232 tablet_replica.cc:330] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:33.035189 1232 raft_consensus.cc:2229] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:33.035559 1232 raft_consensus.cc:2258] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:33.036839 1232 ts_tablet_manager.cc:1875] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:33.039048 29315 consensus_queue.cc:237] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:33.041401 31257 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:33.041940 841 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:33.044046 841 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.048022 28899 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 432dc9a199294b748414520a2c49a674 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:33.048993 31255 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.054268 28913 catalog_manager.cc:5484] T 432dc9a199294b748414520a2c49a674 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.065161 29432 tablet_service.cc:1514] Processing DeleteTablet for tablet 432dc9a199294b748414520a2c49a674 with delete_type TABLET_DATA_TOMBSTONED (TS 139e612d62744c02bcb5653b7aa1dc29 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57020
I20240502 14:16:33.066072 1155 raft_consensus.cc:1050] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:33.068091 30992 raft_consensus.cc:1261] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:33.068682 889 consensus_queue.cc:1035] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:33.069350 1155 consensus_queue.cc:237] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:33.070536 30992 raft_consensus.cc:2932] T 432dc9a199294b748414520a2c49a674 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
W20240502 14:16:33.094439 29386 consensus_peers.cc:485] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6eb9dee735a4115874ef93636a559ab. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:33.096378 29185 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:33.097035 643 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:33.112804 31255 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:33.113381 1153 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:33.117441 29189 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:33.118140 1155 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:33.123129 1079 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:33.123911 29188 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:33.130622 28910 catalog_manager.cc:5484] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:33.133051 31124 raft_consensus.cc:1261] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:33.141512 1155 consensus_queue.cc:1035] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20240502 14:16:33.162017 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.173889 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:33.174736 31388 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:33.175374 643 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:33.176470 1153 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.180384 31255 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.209954 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.265293 28908 catalog_manager.cc:5484] T 3901ce6537fd48ba861fba13767d8aa4 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.268787 31125 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.269560 29189 raft_consensus.cc:2932] T 3901ce6537fd48ba861fba13767d8aa4 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.290176 31388 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:33.303386 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:33.303727 1232 ts_tablet_manager.cc:1888] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:16:33.303901 1232 log.cc:1196] T 3fde0741532a4067a4806f4bc0b0d465 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/3fde0741532a4067a4806f4bc0b0d465
I20240502 14:16:33.309434 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.311635 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet 3fde0741532a4067a4806f4bc0b0d465 (table rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]) successfully deleted
I20240502 14:16:33.320677 1232 tablet_replica.cc:330] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: stopping tablet replica
I20240502 14:16:33.321022 1194 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:33.321472 1194 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:33.321810 1194 ts_tablet_manager.cc:1367] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.317s user 0.059s sys 0.053s
I20240502 14:16:33.322643 1194 raft_consensus.cc:348] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:33.323015 1194 raft_consensus.cc:729] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:33.323227 1194 consensus_queue.cc:260] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:33.323828 1194 ts_tablet_manager.cc:1398] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:33.323884 1232 raft_consensus.cc:2229] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:33.324841 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-e6a0a881fcec43b291148f071e4474fc received from {username='slave'} at 127.21.58.69:59701
I20240502 14:16:33.331257 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-e6a0a881fcec43b291148f071e4474fc on tablet e6a0a881fcec43b291148f071e4474fc with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:33.332908 1232 raft_consensus.cc:2258] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:33.335822 1232 ts_tablet_manager.cc:1875] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:33.337196 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:33.342365 1229 tablet_copy_client.cc:308] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:33.343425 29208 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet e66f9bb135bf4007b1864b63a389fac3 from peer 139e612d62744c02bcb5653b7aa1dc29 ({username='slave'} at 127.21.58.68:45613)
I20240502 14:16:33.343578 29208 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet e66f9bb135bf4007b1864b63a389fac3 from peer 139e612d62744c02bcb5653b7aa1dc29 at {username='slave'} at 127.21.58.68:45613: session id = 139e612d62744c02bcb5653b7aa1dc29-e66f9bb135bf4007b1864b63a389fac3
I20240502 14:16:33.346571 29208 tablet_copy_source_session.cc:205] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:33.349030 1229 ts_tablet_manager.cc:1875] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:16:33.438465 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.457629 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:33.466861 355 raft_consensus.cc:1050] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:33.467832 355 consensus_queue.cc:237] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:33.468778 1229 ts_tablet_manager.cc:1888] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:16:33.470687 1229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f9bb135bf4007b1864b63a389fac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:33.472281 1232 ts_tablet_manager.cc:1888] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:16:33.472425 1232 log.cc:1196] T 432dc9a199294b748414520a2c49a674 P 139e612d62744c02bcb5653b7aa1dc29: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/432dc9a199294b748414520a2c49a674
I20240502 14:16:33.473594 28901 catalog_manager.cc:4830] TS 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): tablet 432dc9a199294b748414520a2c49a674 (table rebalance_test_table_9 [id=c8d2fd0bad3e4c45862835f106e6b44b]) successfully deleted
I20240502 14:16:33.473865 1229 tablet_copy_client.cc:783] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:33.474124 1229 tablet_copy_client.cc:647] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:33.499809 1229 tablet_copy_client.cc:523] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:33.503368 31255 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:33.503976 643 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:33.510996 31125 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:33.511864 643 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:33.513388 29189 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:33.514147 355 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:33.544276 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.544615 643 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.547843 31258 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.556821 28908 catalog_manager.cc:5484] T d95b84899c994d45a569d5a4dda124ed P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.559433 29189 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.570880 1278 ts_tablet_manager.cc:902] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:33.573655 31124 raft_consensus.cc:2932] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:33.591889 1229 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:33.639539 1278 tablet_copy_client.cc:308] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:33.640295 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet a6eb9dee735a4115874ef93636a559ab from peer b7dacf388234446b9a11ce750978ca75 ({username='slave'} at 127.21.58.71:54103)
I20240502 14:16:33.640497 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet a6eb9dee735a4115874ef93636a559ab from peer b7dacf388234446b9a11ce750978ca75 at {username='slave'} at 127.21.58.71:54103: session id = b7dacf388234446b9a11ce750978ca75-a6eb9dee735a4115874ef93636a559ab
I20240502 14:16:33.643357 29472 tablet_copy_source_session.cc:205] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:33.644997 1278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eb9dee735a4115874ef93636a559ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:33.664906 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:33.683881 1278 tablet_copy_client.cc:783] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:33.684176 1278 tablet_copy_client.cc:647] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:33.687330 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:33.693295 1278 tablet_copy_client.cc:523] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:33.694742 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.720796 1278 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: Bootstrap starting.
I20240502 14:16:33.724232 1121 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:33.724680 1121 tablet_bootstrap.cc:492] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:33.725422 1121 ts_tablet_manager.cc:1367] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 1.039s user 0.063s sys 0.006s
I20240502 14:16:33.726231 1121 raft_consensus.cc:348] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:33.726526 1121 raft_consensus.cc:729] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:33.726755 1121 consensus_queue.cc:260] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:33.728812 1121 ts_tablet_manager.cc:1398] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:33.729436 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-39835ac7b2bb4c0a8c4b43b010b09706 received from {username='slave'} at 127.21.58.70:60009
I20240502 14:16:33.729586 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-39835ac7b2bb4c0a8c4b43b010b09706 on tablet 39835ac7b2bb4c0a8c4b43b010b09706 with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:33.753312 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:33.759240 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.771070 31238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.795893 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:33.817041 30992 raft_consensus.cc:1203] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:16:33.821154 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:33.845479 29057 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
W20240502 14:16:33.872062 28990 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:33.873373 31255 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:33.874195 1141 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:33.877103 29315 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:33.877646 1141 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:33.883414 1155 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:33.884168 31255 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:33.890134 28908 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:33.941046 29316 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:33.962460 1229 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:33.962893 1229 tablet_bootstrap.cc:492] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap complete.
I20240502 14:16:33.963184 1229 ts_tablet_manager.cc:1367] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.371s user 0.053s sys 0.005s
I20240502 14:16:33.968636 1229 raft_consensus.cc:348] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:33.969013 1229 raft_consensus.cc:729] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: LEARNER
I20240502 14:16:33.969252 1229 consensus_queue.cc:260] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:33.985232 1229 ts_tablet_manager.cc:1398] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.022s user 0.002s sys 0.001s
I20240502 14:16:33.987847 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session 139e612d62744c02bcb5653b7aa1dc29-e66f9bb135bf4007b1864b63a389fac3 received from {username='slave'} at 127.21.58.68:45613
I20240502 14:16:33.987977 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session 139e612d62744c02bcb5653b7aa1dc29-e66f9bb135bf4007b1864b63a389fac3 on tablet e66f9bb135bf4007b1864b63a389fac3 with peer 139e612d62744c02bcb5653b7aa1dc29
I20240502 14:16:34.010267 29189 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:34.040163 30992 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:34.040931 1216 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:34.048821 31255 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:34.049643 1216 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:34.062657 29057 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:34.063411 1216 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:34.086613 1216 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.087375 29057 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.092244 30992 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.093761 31255 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.094028 28915 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.112396 1278 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:34.112879 1278 tablet_bootstrap.cc:492] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: Bootstrap complete.
I20240502 14:16:34.113163 1278 ts_tablet_manager.cc:1367] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: Time spent bootstrapping tablet: real 0.392s user 0.060s sys 0.000s
I20240502 14:16:34.113116 29189 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:34.113955 1278 raft_consensus.cc:348] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:34.114204 1278 raft_consensus.cc:729] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7dacf388234446b9a11ce750978ca75, State: Initialized, Role: LEARNER
I20240502 14:16:34.114384 1278 consensus_queue.cc:260] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:34.115109 1278 ts_tablet_manager.cc:1398] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:34.116380 29472 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session b7dacf388234446b9a11ce750978ca75-a6eb9dee735a4115874ef93636a559ab received from {username='slave'} at 127.21.58.71:54103
I20240502 14:16:34.116564 29472 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session b7dacf388234446b9a11ce750978ca75-a6eb9dee735a4115874ef93636a559ab on tablet a6eb9dee735a4115874ef93636a559ab with peer b7dacf388234446b9a11ce750978ca75
I20240502 14:16:34.120234 30992 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.120951 681 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:34.123948 681 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.128173 29451 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:34.128535 30990 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.132406 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 981ec6067d624100bddba4383b417c50 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:34.136476 31255 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.137059 1216 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:34.137121 28908 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 9 to 10, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.138625 31255 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.143172 29037 tablet_service.cc:1514] Processing DeleteTablet for tablet 981ec6067d624100bddba4383b417c50 with delete_type TABLET_DATA_TOMBSTONED (TS 9329e8a7a2d1453d90b76b5965315837 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43102
I20240502 14:16:34.146570 31255 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.147182 1079 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:34.147483 31125 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.148669 1079 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:16:34.150930 29385 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302978e1f6034b3693b3a0abacdc494e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.155961 1079 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:34.159998 31255 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:34.167270 31124 raft_consensus.cc:1203] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:34.165418 28908 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:34.177757 29189 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:34.181030 30992 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:34.181612 31255 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:34.181718 1216 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:34.183825 1216 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:16:34.184311 29124 consensus_peers.cc:485] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 981ec6067d624100bddba4383b417c50. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.185536 1216 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:34.192090 30992 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:34.194180 28902 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 981ec6067d624100bddba4383b417c50 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:16:34.193365 31255 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:34.200484 28908 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 10 to 11, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:34.228897 1358 tablet_replica.cc:330] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: stopping tablet replica
I20240502 14:16:34.229256 1358 raft_consensus.cc:2229] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:34.229697 1358 raft_consensus.cc:2258] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:34.231050 1358 ts_tablet_manager.cc:1875] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:34.231609 31124 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:34.249189 29057 consensus_queue.cc:237] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:34.251587 1358 ts_tablet_manager.cc:1888] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:16:34.252123 1358 log.cc:1196] T 981ec6067d624100bddba4383b417c50 P 9329e8a7a2d1453d90b76b5965315837: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/981ec6067d624100bddba4383b417c50
I20240502 14:16:34.253306 28901 catalog_manager.cc:4830] TS 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): tablet 981ec6067d624100bddba4383b417c50 (table rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]) successfully deleted
I20240502 14:16:34.259689 30992 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:16:34.260540 28991 consensus_peers.cc:485] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a230503874a348f4a19647179fe25123. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.260998 1141 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.262666 29189 raft_consensus.cc:1261] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.263273 1141 consensus_queue.cc:1035] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.265329 1153 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:34.266949 30992 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:34.267498 29057 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:34.271445 28910 catalog_manager.cc:5484] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:34.272097 28990 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.273630 31255 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.273641 29315 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.276315 1153 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:34.278120 1141 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:16:34.278846 28993 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fb0fc5919d42d7bedd770945935b42. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.281636 1141 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:34.296577 1079 raft_consensus.cc:1050] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:34.297554 1079 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:34.299353 31255 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:34.301580 29315 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:34.311476 28908 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 9 to 10, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:34.336098 30990 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.336825 355 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:34.340054 29189 raft_consensus.cc:2932] T a230503874a348f4a19647179fe25123 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:34.366778 31255 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.367816 1387 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:34.367930 29315 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:34.370532 1387 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:34.373610 1079 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.379848 28915 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 9 to 10, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:34.378063 30990 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.384955 31257 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.389322 29316 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.395409 29451 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } }
I20240502 14:16:34.404997 31125 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:16:34.405300 29387 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): Couldn't send request to peer 9329e8a7a2d1453d90b76b5965315837. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.405603 29054 consensus_queue.cc:237] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:34.406025 643 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:34.408164 29315 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:34.409201 643 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:34.416687 29451 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:34.419991 31388 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.420651 1155 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.422451 31255 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:34.423166 355 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:16:34.423622 28992 consensus_peers.cc:485] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 -> Peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509): Couldn't send request to peer 139e612d62744c02bcb5653b7aa1dc29. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.424146 29316 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.424784 1155 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.426584 355 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.431985 31257 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.436879 29452 raft_consensus.cc:1203] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:34.441751 30990 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:34.442322 1387 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:34.442855 28915 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 10 to 11, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.444013 30990 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:34.447995 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6a0a881fcec43b291148f071e4474fc with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:34.451527 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a0a881fcec43b291148f071e4474fc with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:34.469630 1413 tablet_replica.cc:330] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:34.470019 1413 raft_consensus.cc:2229] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:34.470641 1413 raft_consensus.cc:2258] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:34.473388 1155 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:34.478957 31391 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:34.486955 355 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } }
I20240502 14:16:34.491552 31124 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } }
I20240502 14:16:34.492647 28915 catalog_manager.cc:5484] T 0272c80ea41149f895074e0e16180b22 P d653f68cb53b4ed79d5e688987ed8d4c reported cstate change: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:34.497952 28915 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 8 to 9, NON_VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:34.498708 1413 ts_tablet_manager.cc:1875] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:34.503643 29315 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } }
I20240502 14:16:34.506781 1413 ts_tablet_manager.cc:1888] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 14:16:34.506980 1413 log.cc:1196] T e6a0a881fcec43b291148f071e4474fc P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6a0a881fcec43b291148f071e4474fc
I20240502 14:16:34.503695 29314 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } } }
I20240502 14:16:34.509035 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet e6a0a881fcec43b291148f071e4474fc (table rebalance_test_table_0 [id=88709dc4b0224a92a8aa5ffdaf78ebf8]) successfully deleted
I20240502 14:16:34.520464 31257 raft_consensus.cc:1203] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:34.574379 29054 consensus_queue.cc:237] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:34.592263 31257 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.592537 29189 raft_consensus.cc:1261] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.593413 1153 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.594120 1440 ts_tablet_manager.cc:902] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753)
I20240502 14:16:34.593986 1415 consensus_queue.cc:1035] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:34.596527 28993 consensus_peers.cc:485] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 239ee601d297412494cf2b447ef75e9a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.597769 29452 consensus_queue.cc:237] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:34.598939 1440 tablet_copy_client.cc:308] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:44753
I20240502 14:16:34.599611 29208 tablet_copy_service.cc:140] P dbabfd51061b4e509d907a3622febd2e: Received BeginTabletCopySession request for tablet 981ec6067d624100bddba4383b417c50 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:45939)
I20240502 14:16:34.599742 29208 tablet_copy_service.cc:161] P dbabfd51061b4e509d907a3622febd2e: Beginning new tablet copy session on tablet 981ec6067d624100bddba4383b417c50 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:45939: session id = 3bb852e0a21446bbbe824bdefb16036c-981ec6067d624100bddba4383b417c50
I20240502 14:16:34.599586 1415 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:34.602385 29208 tablet_copy_source_session.cc:205] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:34.606070 1440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981ec6067d624100bddba4383b417c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:34.606765 28908 catalog_manager.cc:5484] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:34.615885 29189 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:34.621743 30992 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.624071 29189 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.624184 643 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.624605 1387 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:34.627871 29387 consensus_peers.cc:485] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819): Couldn't send request to peer 3bb852e0a21446bbbe824bdefb16036c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63690fff28c84670b15ed02ba1356e33. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.639245 1387 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:34.647105 29185 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:34.652750 1455 ts_tablet_manager.cc:902] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:34.653935 1440 tablet_copy_client.cc:783] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:34.654173 1440 tablet_copy_client.cc:647] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:34.654667 28915 catalog_manager.cc:5484] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:34.655890 1440 tablet_copy_client.cc:523] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:34.662212 31257 raft_consensus.cc:2932] T 239ee601d297412494cf2b447ef75e9a P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:34.676096 30990 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } } }
I20240502 14:16:34.681598 1440 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
I20240502 14:16:34.706676 1474 ts_tablet_manager.cc:902] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:34.720075 29452 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: true } }
W20240502 14:16:34.723675 29388 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): Couldn't send request to peer dbabfd51061b4e509d907a3622febd2e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.724690 31257 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:34.724814 30992 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:34.725368 355 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:34.725631 1387 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:34.729452 1455 tablet_copy_client.cc:308] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:34.730089 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 302978e1f6034b3693b3a0abacdc494e from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:59701)
I20240502 14:16:34.730213 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 302978e1f6034b3693b3a0abacdc494e from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:59701: session id = ede5e1f11f254824a335e1e43f40b6d1-302978e1f6034b3693b3a0abacdc494e
I20240502 14:16:34.733214 29472 tablet_copy_source_session.cc:205] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:34.735626 1415 raft_consensus.cc:1050] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:34.735972 1455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302978e1f6034b3693b3a0abacdc494e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:34.736681 1415 consensus_queue.cc:237] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:34.745325 31255 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: true } } }
I20240502 14:16:34.752804 28908 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P b7dacf388234446b9a11ce750978ca75 reported cstate change: config changed from index 11 to 12, NON_VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: true } } }
I20240502 14:16:34.753276 1479 ts_tablet_manager.cc:902] T 96fb0fc5919d42d7bedd770945935b42 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
I20240502 14:16:34.744524 355 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: true } } }
I20240502 14:16:34.770753 29452 consensus_queue.cc:237] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:34.774861 30992 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: true } } }
I20240502 14:16:34.788473 31125 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:34.789264 1414 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:34.801453 29185 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:34.802012 1414 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:34.810389 30990 raft_consensus.cc:1261] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:34.811218 1414 consensus_queue.cc:1035] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:34.832361 1455 tablet_copy_client.cc:783] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:34.832864 1455 tablet_copy_client.cc:647] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:34.835041 1455 tablet_copy_client.cc:523] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:34.841655 1455 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
W20240502 14:16:34.846462 29385 consensus_peers.cc:485] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 366dbe98fccd42cb8e369876230984fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:34.847422 1414 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:34.848373 29314 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.849584 355 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.851873 31125 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:34.852416 643 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:34.852490 29189 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:34.852969 643 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:34.856359 29314 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:34.860014 28915 catalog_manager.cc:5484] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:34.861660 28908 catalog_manager.cc:5484] T 39835ac7b2bb4c0a8c4b43b010b09706 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:34.868499 31124 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:34.879763 30992 raft_consensus.cc:2932] T 39835ac7b2bb4c0a8c4b43b010b09706 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:34.897848 1216 raft_consensus.cc:1050] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 to VOTER
I20240502 14:16:34.898903 1216 consensus_queue.cc:237] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } }
I20240502 14:16:34.914891 1503 ts_tablet_manager.cc:902] T a230503874a348f4a19647179fe25123 P b7dacf388234446b9a11ce750978ca75: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
F20240502 14:16:34.915369 1503 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-078ffca14316466aa3906d2ff0567c70.queue_time_us, tablet-download-pool-078ffca14316466aa3906d2ff0567c70.queue_time_us) (tablet-download-pool-078ffca14316466aa3906d2ff0567c70.run_cpu_time_us, tablet-download-pool-078ffca14316466aa3906d2ff0567c70.run_cpu_time_us) (tablet-download-pool-078ffca14316466aa3906d2ff0567c70.run_wall_time_us, tablet-download-pool-078ffca14316466aa3906d2ff0567c70.run_wall_time_us) (tablet-download-pool-239ee601d297412494cf2b447ef75e9a.queue_time_us, tablet-download-pool-239ee601d297412494cf2b447ef75e9a.queue_time_us) (tablet-download-pool-239ee601d297412494cf2b447ef75e9a.run_cpu_time_us, tablet-download-pool-239ee601d297412494cf2b447ef75e9a.run_cpu_time_us) (tablet-download-pool-239ee601d297412494cf2b447ef75e9a.run_wall_time_us, tablet-download-pool-239ee601d297412494cf2b447ef75e9a.run_wall_time_us) (tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.queue_time_us, tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.queue_time_us) (tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_cpu_time_us, tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_cpu_time_us) (tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_wall_time_us, tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_wall_time_us) (tablet-download-pool-3901ce6537fd48ba861fba13767d8aa4.queue_time_us, tablet-download-pool-3901ce6537fd48ba861fba13767d8aa4.queue_time_us) (tablet-download-pool-3901ce6537fd48ba861fba13767d8aa4.run_cpu_time_us, tablet-download-pool-3901ce6537fd48ba861fba13767d8aa4.run_cpu_time_us) (tablet-download-pool-3901ce6537fd48ba861fba13767d8aa4.run_wall_time_us, tablet-download-pool-3901ce6537fd48ba861fba13767d8aa4.run_wall_time_us) (tablet-download-pool-432dc9a199294b748414520a2c49a674.queue_time_us, tablet-download-pool-432dc9a199294b748414520a2c49a674.queue_time_us) (tablet-download-pool-432dc9a199294b748414520a2c49a674.run_cpu_time_us, tablet-download-pool-432dc9a199294b748414520a2c49a674.run_cpu_time_us) (tablet-download-pool-432dc9a199294b748414520a2c49a674.run_wall_time_us, tablet-download-pool-432dc9a199294b748414520a2c49a674.run_wall_time_us) (tablet-download-pool-71f047f74165489bacb4f72ab981b447.queue_time_us, tablet-download-pool-71f047f74165489bacb4f72ab981b447.queue_time_us) (tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_cpu_time_us, tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_cpu_time_us) (tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_wall_time_us, tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_wall_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.queue_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.queue_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_cpu_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_cpu_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_wall_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_wall_time_us) (tablet-download-pool-981ec6067d624100bddba4383b417c50.queue_time_us, tablet-download-pool-981ec6067d624100bddba4383b417c50.queue_time_us) (tablet-download-pool-981ec6067d624100bddba4383b417c50.run_cpu_time_us, tablet-download-pool-981ec6067d624100bddba4383b417c50.run_cpu_time_us) (tablet-download-pool-981ec6067d624100bddba4383b417c50.run_wall_time_us, tablet-download-pool-981ec6067d624100bddba4383b417c50.run_wall_time_us) (tablet-download-pool-a230503874a348f4a19647179fe25123.queue_time_us, tablet-download-pool-a230503874a348f4a19647179fe25123.queue_time_us) (tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.queue_time_us, tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.queue_time_us) (tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_cpu_time_us, tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_cpu_time_us) (tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_wall_time_us, tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_wall_time_us) (tablet-download-pool-c4818cd7d9b44111a04a0d32a4580cdf.queue_time_us, tablet-download-pool-c4818cd7d9b44111a04a0d32a4580cdf.queue_time_us) (tablet-download-pool-c4818cd7d9b44111a04a0d32a4580cdf.run_cpu_time_us, tablet-download-pool-c4818cd7d9b44111a04a0d32a4580cdf.run_cpu_time_us) (tablet-download-pool-c4818cd7d9b44111a04a0d32a4580cdf.run_wall_time_us, tablet-download-pool-c4818cd7d9b44111a04a0d32a4580cdf.run_wall_time_us) (tablet-download-pool-d95b84899c994d45a569d5a4dda124ed.queue_time_us, tablet-download-pool-d95b84899c994d45a569d5a4dda124ed.queue_time_us) (tablet-download-pool-d95b84899c994d45a569d5a4dda124ed.run_cpu_time_us, tablet-download-pool-d95b84899c994d45a569d5a4dda124ed.run_cpu_time_us) (tablet-download-pool-d95b84899c994d45a569d5a4dda124ed.run_wall_time_us, tablet-download-pool-d95b84899c994d45a569d5a4dda124ed.run_wall_time_us) (tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.queue_time_us, tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.queue_time_us) (tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_cpu_time_us, tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_cpu_time_us) (tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_wall_time_us, tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_wall_time_us) (tablet-download-pool-e89c7a017200449abcd5d9b9c2bcc912.queue_time_us, tablet-download-pool-e89c7a017200449abcd5d9b9c2bcc912.queue_time_us) (tablet-download-pool-e89c7a017200449abcd5d9b9c2bcc912.run_cpu_time_us, tablet-download-pool-e89c7a017200449abcd5d9b9c2bcc912.run_cpu_time_us) (tablet-download-pool-e89c7a017200449abcd5d9b9c2bcc912.run_wall_time_us, tablet-download-pool-e89c7a017200449abcd5d9b9c2bcc912.run_wall_time_us) (tablet-download-pool-fd4e4241d42d483ab6203606a4d92141.queue_time_us, tablet-download-pool-fd4e4241d42d483ab6203606a4d92141.queue_time_us) (tablet-download-pool-fd4e4241d42d483ab6203606a4d92141.run_cpu_time_us, tablet-download-pool-fd4e4241d42d483ab6203606a4d92141.run_cpu_time_us) (tablet-download-pool-fd4e4241d42d483ab6203606a4d92141.run_wall_time_us, tablet-download-pool-fd4e4241d42d483ab6203606a4d92141.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:16:34.922255 31125 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:34.930202 29451 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:34.930902 1364 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:34.931452 1479 tablet_copy_client.cc:308] T 96fb0fc5919d42d7bedd770945935b42 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:34.932230 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 96fb0fc5919d42d7bedd770945935b42 from peer d653f68cb53b4ed79d5e688987ed8d4c ({username='slave'} at 127.21.58.72:39523)
I20240502 14:16:34.932425 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 96fb0fc5919d42d7bedd770945935b42 from peer d653f68cb53b4ed79d5e688987ed8d4c at {username='slave'} at 127.21.58.72:39523: session id = d653f68cb53b4ed79d5e688987ed8d4c-96fb0fc5919d42d7bedd770945935b42
I20240502 14:16:34.936978 29077 tablet_copy_source_session.cc:205] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:34.943218 31388 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:34.943539 29054 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:34.944393 681 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:34.945350 681 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:34.953286 1479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fb0fc5919d42d7bedd770945935b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:34.970417 1530 ts_tablet_manager.cc:902] T 239ee601d297412494cf2b447ef75e9a P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
F20240502 14:16:34.970948 1530 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0272c80ea41149f895074e0e16180b22.queue_time_us, tablet-download-pool-0272c80ea41149f895074e0e16180b22.queue_time_us) (tablet-download-pool-0272c80ea41149f895074e0e16180b22.run_cpu_time_us, tablet-download-pool-0272c80ea41149f895074e0e16180b22.run_cpu_time_us) (tablet-download-pool-0272c80ea41149f895074e0e16180b22.run_wall_time_us, tablet-download-pool-0272c80ea41149f895074e0e16180b22.run_wall_time_us) (tablet-download-pool-093e29d30b2646d68e84b8b9ca86ca9e.queue_time_us, tablet-download-pool-093e29d30b2646d68e84b8b9ca86ca9e.queue_time_us) (tablet-download-pool-093e29d30b2646d68e84b8b9ca86ca9e.run_cpu_time_us, tablet-download-pool-093e29d30b2646d68e84b8b9ca86ca9e.run_cpu_time_us) (tablet-download-pool-093e29d30b2646d68e84b8b9ca86ca9e.run_wall_time_us, tablet-download-pool-093e29d30b2646d68e84b8b9ca86ca9e.run_wall_time_us) (tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.queue_time_us, tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.queue_time_us) (tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_cpu_time_us, tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_cpu_time_us) (tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_wall_time_us, tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_wall_time_us) (tablet-download-pool-239ee601d297412494cf2b447ef75e9a.queue_time_us, tablet-download-pool-239ee601d297412494cf2b447ef75e9a.queue_time_us) (tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.queue_time_us, tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.queue_time_us) (tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_cpu_time_us, tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_cpu_time_us) (tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_wall_time_us, tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_wall_time_us) (tablet-download-pool-44d94461e22c4ae182afa2a15f3db198.queue_time_us, tablet-download-pool-44d94461e22c4ae182afa2a15f3db198.queue_time_us) (tablet-download-pool-44d94461e22c4ae182afa2a15f3db198.run_cpu_time_us, tablet-download-pool-44d94461e22c4ae182afa2a15f3db198.run_cpu_time_us) (tablet-download-pool-44d94461e22c4ae182afa2a15f3db198.run_wall_time_us, tablet-download-pool-44d94461e22c4ae182afa2a15f3db198.run_wall_time_us) (tablet-download-pool-71f047f74165489bacb4f72ab981b447.queue_time_us, tablet-download-pool-71f047f74165489bacb4f72ab981b447.queue_time_us) (tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_cpu_time_us, tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_cpu_time_us) (tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_wall_time_us, tablet-download-pool-71f047f74165489bacb4f72ab981b447.run_wall_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.queue_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.queue_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_cpu_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_cpu_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_wall_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_wall_time_us) (tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.queue_time_us, tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.queue_time_us) (tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_cpu_time_us, tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_cpu_time_us) (tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_wall_time_us, tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_wall_time_us) (tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.queue_time_us, tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.queue_time_us) (tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_cpu_time_us, tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_cpu_time_us) (tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_wall_time_us, tablet-download-pool-a6eb9dee735a4115874ef93636a559ab.run_wall_time_us) (tablet-download-pool-b7bfc3bb327b4407ae1aa7742f4b18de.queue_time_us, tablet-download-pool-b7bfc3bb327b4407ae1aa7742f4b18de.queue_time_us) (tablet-download-pool-b7bfc3bb327b4407ae1aa7742f4b18de.run_cpu_time_us, tablet-download-pool-b7bfc3bb327b4407ae1aa7742f4b18de.run_cpu_time_us) (tablet-download-pool-b7bfc3bb327b4407ae1aa7742f4b18de.run_wall_time_us, tablet-download-pool-b7bfc3bb327b4407ae1aa7742f4b18de.run_wall_time_us) (tablet-download-pool-d3221d1a1b244a27abc9ee26e0a6edb1.queue_time_us, tablet-download-pool-d3221d1a1b244a27abc9ee26e0a6edb1.queue_time_us) (tablet-download-pool-d3221d1a1b244a27abc9ee26e0a6edb1.run_cpu_time_us, tablet-download-pool-d3221d1a1b244a27abc9ee26e0a6edb1.run_cpu_time_us) (tablet-download-pool-d3221d1a1b244a27abc9ee26e0a6edb1.run_wall_time_us, tablet-download-pool-d3221d1a1b244a27abc9ee26e0a6edb1.run_wall_time_us) (tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.queue_time_us, tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.queue_time_us) (tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_cpu_time_us, tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_cpu_time_us) (tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_wall_time_us, tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_wall_time_us) (tablet-download-pool-e66f9bb135bf4007b1864b63a389fac3.queue_time_us, tablet-download-pool-e66f9bb135bf4007b1864b63a389fac3.queue_time_us) (tablet-download-pool-e66f9bb135bf4007b1864b63a389fac3.run_cpu_time_us, tablet-download-pool-e66f9bb135bf4007b1864b63a389fac3.run_cpu_time_us) (tablet-download-pool-e66f9bb135bf4007b1864b63a389fac3.run_wall_time_us, tablet-download-pool-e66f9bb135bf4007b1864b63a389fac3.run_wall_time_us) (tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.queue_time_us, tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.queue_time_us) (tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.run_cpu_time_us, tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.run_cpu_time_us) (tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.run_wall_time_us, tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f9a6582bdcd google::LogMessage::Fail() at ??:0
I20240502 14:16:34.984858 1364 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:34.991466 31388 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:34.992031 28910 catalog_manager.cc:5484] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7fc3149d8dcd google::LogMessage::Fail() at ??:0
I20240502 14:16:34.995734 29451 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:35.000348 29185 consensus_queue.cc:237] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } }
I20240502 14:16:34.995198 29054 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
@ 0x7f9a6582fb93 google::LogMessage::SendToLog() at ??:0
I20240502 14:16:35.018761 1479 tablet_copy_client.cc:783] T 96fb0fc5919d42d7bedd770945935b42 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:35.019902 1479 tablet_copy_client.cc:647] T 96fb0fc5919d42d7bedd770945935b42 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:35.031320 1474 tablet_copy_client.cc:308] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:35.032183 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 96fb0fc5919d42d7bedd770945935b42 from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:48855)
I20240502 14:16:35.032392 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 96fb0fc5919d42d7bedd770945935b42 from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:48855: session id = ede5e1f11f254824a335e1e43f40b6d1-96fb0fc5919d42d7bedd770945935b42
I20240502 14:16:35.036420 29077 tablet_copy_source_session.cc:205] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:35.037693 1474 ts_tablet_manager.cc:1875] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:16:35.038177 1479 tablet_copy_client.cc:523] T 96fb0fc5919d42d7bedd770945935b42 P d653f68cb53b4ed79d5e688987ed8d4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:35.041507 1479 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P d653f68cb53b4ed79d5e688987ed8d4c: Bootstrap starting.
I20240502 14:16:35.043814 1544 ts_tablet_manager.cc:902] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:35.044929 1546 ts_tablet_manager.cc:902] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Initiating tablet copy from peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783)
@ 0x7f9a6582b7cc google::LogMessage::Flush() at ??:0
I20240502 14:16:35.069566 29452 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:35.069875 29057 consensus_queue.cc:237] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }
I20240502 14:16:35.070371 29056 raft_consensus.cc:1261] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
@ 0x7f9a6582cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:16:35.073684 1547 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:35.074291 1547 consensus_queue.cc:1035] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:35.076772 1547 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:35.080227 29056 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
W20240502 14:16:35.081405 28991 consensus_peers.cc:485] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7bfc3bb327b4407ae1aa7742f4b18de. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:35.082742 31388 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:35.083334 1153 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:35.083868 29452 raft_consensus.cc:2932] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:35.084070 29314 consensus_queue.cc:237] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:35.090620 1555 ts_tablet_manager.cc:902] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:35.090468 28910 catalog_manager.cc:5484] T e66f9bb135bf4007b1864b63a389fac3 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 8 to 9, VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
@ 0x7fc3149dcb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc3149d87cc google::LogMessage::Flush() at ??:0
I20240502 14:16:35.095997 1153 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:35.098750 31370 tablet_service.cc:1514] Processing DeleteTablet for tablet e66f9bb135bf4007b1864b63a389fac3 with delete_type TABLET_DATA_TOMBSTONED (TS d653f68cb53b4ed79d5e688987ed8d4c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60838
I20240502 14:16:35.099179 31391 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:35.103008 28902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e66f9bb135bf4007b1864b63a389fac3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:35.104501 29452 raft_consensus.cc:1261] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:35.105047 1415 consensus_queue.cc:1035] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:35.111474 28910 catalog_manager.cc:5484] T b7bfc3bb327b4407ae1aa7742f4b18de P d653f68cb53b4ed79d5e688987ed8d4c reported cstate change: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:35.117008 643 raft_consensus.cc:1050] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER b7dacf388234446b9a11ce750978ca75 to VOTER
I20240502 14:16:35.117825 643 consensus_queue.cc:237] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
W20240502 14:16:35.128790 29257 consensus_peers.cc:485] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a00d9c46bfc64a4085715e4f764b8c96. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:35.130259 30990 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:35.130911 1507 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:35.134497 1440 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:35.135015 1440 tablet_bootstrap.cc:492] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:35.135356 1440 ts_tablet_manager.cc:1367] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 0.454s user 0.067s sys 0.006s
I20240502 14:16:35.136209 1440 raft_consensus.cc:348] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:35.136464 1440 raft_consensus.cc:729] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:35.136665 1440 consensus_queue.cc:260] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:35.137257 1440 ts_tablet_manager.cc:1398] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:35.138038 29208 tablet_copy_service.cc:342] P dbabfd51061b4e509d907a3622febd2e: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-981ec6067d624100bddba4383b417c50 received from {username='slave'} at 127.21.58.70:45939
I20240502 14:16:35.138178 29208 tablet_copy_service.cc:433] P dbabfd51061b4e509d907a3622febd2e: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-981ec6067d624100bddba4383b417c50 on tablet 981ec6067d624100bddba4383b417c50 with peer 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:35.142882 29056 raft_consensus.cc:1261] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 1b3c20f7bfd548769f6bd40148590b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:16:35.143745 1507 consensus_queue.cc:1035] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:16:35.149080 31255 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:35.149829 355 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:35.153115 1474 ts_tablet_manager.cc:1888] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 14:16:35.153522 1474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fb0fc5919d42d7bedd770945935b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:35.156754 31388 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:35.157335 355 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:35.157843 1507 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:35.161188 1562 tablet_replica.cc:330] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: stopping tablet replica
I20240502 14:16:35.161419 1562 raft_consensus.cc:2229] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:35.161729 1562 raft_consensus.cc:2258] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:35.162667 29057 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:35.167908 29189 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:35.168473 355 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:35.169132 1563 ts_tablet_manager.cc:902] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:35.171463 28908 catalog_manager.cc:5484] T a00d9c46bfc64a4085715e4f764b8c96 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New cstate: current_term: 1 leader_uuid: "1b3c20f7bfd548769f6bd40148590b74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:35.172607 30992 raft_consensus.cc:2932] T a00d9c46bfc64a4085715e4f764b8c96 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
@ 0x7f9a6856c742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fc3149d9f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:16:35.184576 1562 ts_tablet_manager.cc:1875] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:35.184304 355 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:35.189707 1546 tablet_copy_client.cc:308] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:46783
I20240502 14:16:35.190420 29077 tablet_copy_service.cc:140] P 9329e8a7a2d1453d90b76b5965315837: Received BeginTabletCopySession request for tablet 0272c80ea41149f895074e0e16180b22 from peer 139e612d62744c02bcb5653b7aa1dc29 ({username='slave'} at 127.21.58.68:39235)
I20240502 14:16:35.190563 29077 tablet_copy_service.cc:161] P 9329e8a7a2d1453d90b76b5965315837: Beginning new tablet copy session on tablet 0272c80ea41149f895074e0e16180b22 from peer 139e612d62744c02bcb5653b7aa1dc29 at {username='slave'} at 127.21.58.68:39235: session id = 139e612d62744c02bcb5653b7aa1dc29-0272c80ea41149f895074e0e16180b22
I20240502 14:16:35.193821 29077 tablet_copy_source_session.cc:205] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:35.196264 31388 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:35.197468 1546 ts_tablet_manager.cc:1875] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:16:35.201575 31255 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
@ 0x7f9a68538f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:16:35.213387 1474 tablet_copy_client.cc:783] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:35.213655 1474 tablet_copy_client.cc:647] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:35.206696 28908 catalog_manager.cc:5484] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:35.216892 1544 tablet_copy_client.cc:308] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:35.217743 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet f4dbd9bc62484dd1998080d289db098f from peer 9329e8a7a2d1453d90b76b5965315837 ({username='slave'} at 127.21.58.65:47501)
I20240502 14:16:35.217803 1474 tablet_copy_client.cc:523] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:35.217917 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet f4dbd9bc62484dd1998080d289db098f from peer 9329e8a7a2d1453d90b76b5965315837 at {username='slave'} at 127.21.58.65:47501: session id = 9329e8a7a2d1453d90b76b5965315837-f4dbd9bc62484dd1998080d289db098f
I20240502 14:16:35.220868 29472 tablet_copy_source_session.cc:205] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:35.229607 1474 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:35.231300 1555 tablet_copy_client.cc:308] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:35.231199 29189 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b7dacf388234446b9a11ce750978ca75 (127.21.58.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:35.232012 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 63690fff28c84670b15ed02ba1356e33 from peer 3bb852e0a21446bbbe824bdefb16036c ({username='slave'} at 127.21.58.70:56885)
I20240502 14:16:35.232156 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 63690fff28c84670b15ed02ba1356e33 from peer 3bb852e0a21446bbbe824bdefb16036c at {username='slave'} at 127.21.58.70:56885: session id = 3bb852e0a21446bbbe824bdefb16036c-63690fff28c84670b15ed02ba1356e33
I20240502 14:16:35.235184 29472 tablet_copy_source_session.cc:205] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:35.239341 1544 ts_tablet_manager.cc:1875] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:16:35.246176 29452 consensus_queue.cc:237] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:35.249483 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63690fff28c84670b15ed02ba1356e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:35.252430 31257 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
@ 0x7f9a68532cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:16:35.253168 355 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:35.259940 355 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:35.265800 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6eb9dee735a4115874ef93636a559ab with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:35.268289 31257 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P b7dacf388234446b9a11ce750978ca75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
I20240502 14:16:35.272850 1546 ts_tablet_manager.cc:1888] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:16:35.273149 1546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0272c80ea41149f895074e0e16180b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:35.278963 1562 ts_tablet_manager.cc:1888] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:16:35.279140 1562 log.cc:1196] T e66f9bb135bf4007b1864b63a389fac3 P d653f68cb53b4ed79d5e688987ed8d4c: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-7/wal/wals/e66f9bb135bf4007b1864b63a389fac3
I20240502 14:16:35.280227 28902 catalog_manager.cc:4830] TS d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): tablet e66f9bb135bf4007b1864b63a389fac3 (table rebalance_test_table_2 [id=97eb0e50641648b8b13df0a289a86046]) successfully deleted
I20240502 14:16:35.282497 31388 raft_consensus.cc:1261] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:16:35.283031 355 consensus_queue.cc:1035] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:35.284493 31391 raft_consensus.cc:2932] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
@ 0x7fc317719742 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:16:35.312561 1544 ts_tablet_manager.cc:1888] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:16:35.312899 1544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dbd9bc62484dd1998080d289db098f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:35.321499 1608 ts_tablet_manager.cc:902] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
I20240502 14:16:35.322961 28910 catalog_manager.cc:5484] T a6eb9dee735a4115874ef93636a559ab P d653f68cb53b4ed79d5e688987ed8d4c reported cstate change: config changed from index 8 to 9, VOTER dbabfd51061b4e509d907a3622febd2e (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } }
@ 0x7f9a779858de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:16:35.346010 1555 tablet_copy_client.cc:783] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:35.346304 1555 tablet_copy_client.cc:647] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Starting download of 1 WAL segments...
@ 0x7fc3176e5f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:16:35.350000 1608 tablet_copy_client.cc:308] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:35.339475 29451 raft_consensus.cc:2932] T b7bfc3bb327b4407ae1aa7742f4b18de P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } } }
I20240502 14:16:35.353528 29472 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet 366dbe98fccd42cb8e369876230984fe from peer ede5e1f11f254824a335e1e43f40b6d1 ({username='slave'} at 127.21.58.69:59701)
I20240502 14:16:35.353814 29472 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet 366dbe98fccd42cb8e369876230984fe from peer ede5e1f11f254824a335e1e43f40b6d1 at {username='slave'} at 127.21.58.69:59701: session id = ede5e1f11f254824a335e1e43f40b6d1-366dbe98fccd42cb8e369876230984fe
I20240502 14:16:35.357450 1546 tablet_copy_client.cc:783] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:35.357697 1546 tablet_copy_client.cc:647] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Starting download of 1 WAL segments...
@ 0x7f9a7799ddaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:16:35.364873 29472 tablet_copy_source_session.cc:205] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:35.366079 1608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366dbe98fccd42cb8e369876230984fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:35.371572 1546 tablet_copy_client.cc:523] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:35.373531 1555 tablet_copy_client.cc:523] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:35.379099 1608 tablet_copy_client.cc:783] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:35.380177 1608 tablet_copy_client.cc:647] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:35.380465 1563 tablet_copy_client.cc:308] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.68:46509
I20240502 14:16:35.381325 29168 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eb9dee735a4115874ef93636a559ab with delete_type TABLET_DATA_TOMBSTONED (TS dbabfd51061b4e509d907a3622febd2e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39478
I20240502 14:16:35.381521 29471 tablet_copy_service.cc:140] P 139e612d62744c02bcb5653b7aa1dc29: Received BeginTabletCopySession request for tablet e6a0a881fcec43b291148f071e4474fc from peer dbabfd51061b4e509d907a3622febd2e ({username='slave'} at 127.21.58.66:59129)
I20240502 14:16:35.381738 29471 tablet_copy_service.cc:161] P 139e612d62744c02bcb5653b7aa1dc29: Beginning new tablet copy session on tablet e6a0a881fcec43b291148f071e4474fc from peer dbabfd51061b4e509d907a3622febd2e at {username='slave'} at 127.21.58.66:59129: session id = dbabfd51061b4e509d907a3622febd2e-e6a0a881fcec43b291148f071e4474fc
I20240502 14:16:35.382352 1608 tablet_copy_client.cc:523] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:35.387287 29471 tablet_copy_source_session.cc:205] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:35.388990 1563 ts_tablet_manager.cc:1875] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:16:35.393817 1619 tablet_replica.cc:330] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: stopping tablet replica
I20240502 14:16:35.394021 1619 raft_consensus.cc:2229] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:35.394284 1619 raft_consensus.cc:2258] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:35.395406 1608 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap starting.
I20240502 14:16:35.396724 1546 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap starting.
I20240502 14:16:35.397205 1619 ts_tablet_manager.cc:1875] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:35.403923 1544 tablet_copy_client.cc:783] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:35.404181 1544 tablet_copy_client.cc:647] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:35.406911 1555 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap starting.
@ 0x7fc3176dfcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f9a77ac8480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 14:16:35.443935 1544 tablet_copy_client.cc:523] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:35.450520 1563 ts_tablet_manager.cc:1888] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:16:35.450919 1563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a0a881fcec43b291148f071e4474fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:35.455250 1544 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Bootstrap starting.
I20240502 14:16:35.467587 1619 ts_tablet_manager.cc:1888] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:16:35.467751 1619 log.cc:1196] T a6eb9dee735a4115874ef93636a559ab P dbabfd51061b4e509d907a3622febd2e: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6eb9dee735a4115874ef93636a559ab
I20240502 14:16:35.468605 28902 catalog_manager.cc:4830] TS dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): tablet a6eb9dee735a4115874ef93636a559ab (table rebalance_test_table_6 [id=c36807ea6a59441696e14f8d2dbf3053]) successfully deleted
@ 0x7f9a77b00a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f9a77ae3981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f9a77ae370e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9a6853cb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9a6854019c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9a68540019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9a6851b01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a6b8a06db start_thread at ??:0
@ 0x7f9a646de71f clone at ??:0
@ 0x7fc326b328de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:16:35.622961 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:35.628764 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:35.642755 1608 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:35.643411 1608 tablet_bootstrap.cc:492] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:35.643688 1608 ts_tablet_manager.cc:1367] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 0.248s user 0.047s sys 0.008s
I20240502 14:16:35.644539 1608 raft_consensus.cc:348] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:35.644845 1608 raft_consensus.cc:729] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:35.645073 1608 consensus_queue.cc:260] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:35.649638 31125 raft_consensus.cc:1203] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 14:16:35.684996 31107 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:35.686897 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:35.689587 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:35.543028 1563 tablet_copy_client.cc:783] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:35.716957 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:35.720681 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:35.716959 1563 tablet_copy_client.cc:647] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:35.755561 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:35.757555 1608 ts_tablet_manager.cc:1398] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.114s user 0.000s sys 0.002s
I20240502 14:16:35.761893 29471 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-366dbe98fccd42cb8e369876230984fe received from {username='slave'} at 127.21.58.69:59701
I20240502 14:16:35.762084 29471 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-366dbe98fccd42cb8e369876230984fe on tablet 366dbe98fccd42cb8e369876230984fe with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:35.768308 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:35.770393 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:35.801617 1563 tablet_copy_client.cc:523] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:35.816498 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:35.816656 1683 ts_tablet_manager.cc:902] T a00d9c46bfc64a4085715e4f764b8c96 P d653f68cb53b4ed79d5e688987ed8d4c: Initiating tablet copy from peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039)
@ 0x7fc326b4adaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:16:35.847921 1563 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Bootstrap starting.
I20240502 14:16:35.860873 29189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198"
dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51724
I20240502 14:16:35.861174 29189 raft_consensus.cc:595] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:35.870362 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:35.871382 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:35.905793 31370 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:35.926901 30990 raft_consensus.cc:1203] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
@ 0x7fc326c75480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 14:16:36.031380 1547 raft_consensus.cc:979] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e: : Instructing follower 3bb852e0a21446bbbe824bdefb16036c to start an election
I20240502 14:16:36.031559 1547 raft_consensus.cc:1067] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Signalling peer 3bb852e0a21446bbbe824bdefb16036c to start an election
I20240502 14:16:36.032483 31125 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198"
dest_uuid: "3bb852e0a21446bbbe824bdefb16036c"
from {username='slave'} at 127.21.58.66:43931
I20240502 14:16:36.032753 31125 raft_consensus.cc:482] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:36.032945 31125 raft_consensus.cc:3037] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:36.036041 31125 raft_consensus.cc:504] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:36.057056 29189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198" candidate_uuid: "3bb852e0a21446bbbe824bdefb16036c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
I20240502 14:16:36.057303 29189 raft_consensus.cc:3032] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:36.057374 29189 raft_consensus.cc:729] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Running, Role: LEADER
I20240502 14:16:36.057683 29189 consensus_queue.cc:260] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:36.058110 29189 raft_consensus.cc:3037] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:36.061091 29189 raft_consensus.cc:2454] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb852e0a21446bbbe824bdefb16036c in term 2.
I20240502 14:16:36.070538 31125 leader_election.cc:288] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [CANDIDATE]: Term 2 election: Requested vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549)
I20240502 14:16:36.102228 31388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198" candidate_uuid: "3bb852e0a21446bbbe824bdefb16036c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d653f68cb53b4ed79d5e688987ed8d4c"
I20240502 14:16:36.102679 31388 raft_consensus.cc:3037] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 1 FOLLOWER]: Advancing to term 2
@ 0x7fc326cada8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:16:36.103399 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d94461e22c4ae182afa2a15f3db198" candidate_uuid: "3bb852e0a21446bbbe824bdefb16036c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1b3c20f7bfd548769f6bd40148590b74"
I20240502 14:16:36.103598 29314 raft_consensus.cc:3037] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:36.106128 31388 raft_consensus.cc:2454] T 44d94461e22c4ae182afa2a15f3db198 P d653f68cb53b4ed79d5e688987ed8d4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb852e0a21446bbbe824bdefb16036c in term 2.
I20240502 14:16:36.106199 29314 raft_consensus.cc:2454] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb852e0a21446bbbe824bdefb16036c in term 2.
I20240502 14:16:36.106765 31062 leader_election.cc:302] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3bb852e0a21446bbbe824bdefb16036c, d653f68cb53b4ed79d5e688987ed8d4c, dbabfd51061b4e509d907a3622febd2e; no voters:
I20240502 14:16:36.107175 1680 raft_consensus.cc:2781] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:36.132091 1680 raft_consensus.cc:686] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [term 2 LEADER]: Becoming Leader. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Running, Role: LEADER
I20240502 14:16:36.132623 1680 consensus_queue.cc:237] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } }
I20240502 14:16:36.138484 1547 raft_consensus.cc:1050] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:36.139425 1547 consensus_queue.cc:237] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:36.147727 28915 catalog_manager.cc:5484] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: term changed from 1 to 2, leader changed from dbabfd51061b4e509d907a3622febd2e (127.21.58.66) to 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70). New cstate: current_term: 2 leader_uuid: "3bb852e0a21446bbbe824bdefb16036c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:36.150410 30990 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:36.151185 1364 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
@ 0x7fc326c90981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240502 14:16:36.160777 31125 raft_consensus.cc:1261] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:36.162360 1455 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:36.162355 1364 consensus_queue.cc:1035] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:36.162802 1455 tablet_bootstrap.cc:492] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:36.163095 1455 ts_tablet_manager.cc:1367] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 1.322s user 0.055s sys 0.041s
I20240502 14:16:36.165036 1455 raft_consensus.cc:348] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:36.165282 1455 raft_consensus.cc:729] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:36.165510 1455 consensus_queue.cc:260] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:36.167910 1546 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:36.168272 1546 tablet_bootstrap.cc:492] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Bootstrap complete.
I20240502 14:16:36.168517 1546 ts_tablet_manager.cc:1367] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Time spent bootstrapping tablet: real 0.772s user 0.068s sys 0.058s
I20240502 14:16:36.169221 1546 raft_consensus.cc:348] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:36.169446 1546 raft_consensus.cc:729] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Initialized, Role: LEARNER
I20240502 14:16:36.169612 1546 consensus_queue.cc:260] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: true } }
I20240502 14:16:36.171551 1455 ts_tablet_manager.cc:1398] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:16:36.172128 29471 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-302978e1f6034b3693b3a0abacdc494e received from {username='slave'} at 127.21.58.69:59701
I20240502 14:16:36.172245 29471 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-302978e1f6034b3693b3a0abacdc494e on tablet 302978e1f6034b3693b3a0abacdc494e with peer ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:36.181151 1546 ts_tablet_manager.cc:1398] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 14:16:36.237821 1364 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
@ 0x7fc326c9070e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 14:16:36.258132 30990 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:36.262126 31125 raft_consensus.cc:2932] T 981ec6067d624100bddba4383b417c50 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
@ 0x7fc3176e9b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 14:16:36.317423 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session 139e612d62744c02bcb5653b7aa1dc29-0272c80ea41149f895074e0e16180b22 received from {username='slave'} at 127.21.58.68:39235
I20240502 14:16:36.317593 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session 139e612d62744c02bcb5653b7aa1dc29-0272c80ea41149f895074e0e16180b22 on tablet 0272c80ea41149f895074e0e16180b22 with peer 139e612d62744c02bcb5653b7aa1dc29
W20240502 14:16:36.318292 29255 connection.cc:535] client connection to 127.21.58.71:44703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:36.318522 29255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:36.322818 28991 connection.cc:535] client connection to 127.21.58.71:44703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:36.323127 28991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:36.346174 29386 connection.cc:535] client connection to 127.21.58.71:44703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:36.346498 29386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7fc3176ed19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
W20240502 14:16:36.366291 1752 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:36.322784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.58.71:44703 (local address 127.21.58.67:36073)
0502 14:16:36.355658 (+ 32874us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:36.355771 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":32751,"thread_start_us":36297,"threads_started":1}
W20240502 14:16:36.366452 29255 consensus_peers.cc:485] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.366540 29255 consensus_peers.cc:485] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.366631 29255 consensus_peers.cc:485] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:36.369750 1474 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:36.370213 1474 tablet_bootstrap.cc:492] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Bootstrap complete.
I20240502 14:16:36.370510 1474 ts_tablet_manager.cc:1367] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Time spent bootstrapping tablet: real 1.141s user 0.048s sys 0.006s
@ 0x7fc3176ed019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 14:16:36.383406 1474 raft_consensus.cc:348] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:36.383692 1474 raft_consensus.cc:729] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ede5e1f11f254824a335e1e43f40b6d1, State: Initialized, Role: LEARNER
I20240502 14:16:36.383880 1474 consensus_queue.cc:260] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:36.384409 1474 ts_tablet_manager.cc:1398] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:16:36.385018 29077 tablet_copy_service.cc:342] P 9329e8a7a2d1453d90b76b5965315837: Request end of tablet copy session ede5e1f11f254824a335e1e43f40b6d1-96fb0fc5919d42d7bedd770945935b42 received from {username='slave'} at 127.21.58.69:48855
I20240502 14:16:36.385165 29077 tablet_copy_service.cc:433] P 9329e8a7a2d1453d90b76b5965315837: ending tablet copy session ede5e1f11f254824a335e1e43f40b6d1-96fb0fc5919d42d7bedd770945935b42 on tablet 96fb0fc5919d42d7bedd770945935b42 with peer ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:36.385569 29123 connection.cc:535] client connection to 127.21.58.71:44703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:36.386030 29123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7fc3176c801e kudu::Thread::SuperviseThread() at ??:0
W20240502 14:16:36.404227 28991 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.71:44703, user_credentials={real_user=slave}} blocked reactor thread for 81207.1us
@ 0x7fc31aa4d6db start_thread at ??:0
W20240502 14:16:36.406002 29256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.65:46783, user_credentials={real_user=slave}} blocked reactor thread for 81966.7us
@ 0x7fc31388b71f clone at ??:0
I20240502 14:16:36.412200 28913 catalog_manager.cc:5484] T 981ec6067d624100bddba4383b417c50 P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 11 to 12, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:36.418107 1555 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:36.418570 1555 tablet_bootstrap.cc:492] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: Bootstrap complete.
I20240502 14:16:36.418845 1555 ts_tablet_manager.cc:1367] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: Time spent bootstrapping tablet: real 1.012s user 0.048s sys 0.008s
I20240502 14:16:36.419696 1555 raft_consensus.cc:348] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:36.419934 1555 raft_consensus.cc:729] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb852e0a21446bbbe824bdefb16036c, State: Initialized, Role: LEARNER
I20240502 14:16:36.420176 1555 consensus_queue.cc:260] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: true } }
I20240502 14:16:36.422179 1555 ts_tablet_manager.cc:1398] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20240502 14:16:36.433235 29386 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.71:44703, user_credentials={real_user=slave}} blocked reactor thread for 86825.1us
I20240502 14:16:36.437777 355 raft_consensus.cc:1050] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
W20240502 14:16:36.438138 29123 consensus_peers.cc:485] T fd4e4241d42d483ab6203606a4d92141 P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.438360 29123 consensus_peers.cc:485] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.438526 29123 consensus_peers.cc:485] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:36.438616 355 consensus_queue.cc:237] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
W20240502 14:16:36.475132 28991 consensus_peers.cc:462] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
W20240502 14:16:36.475292 28991 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.475412 28991 consensus_peers.cc:462] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
W20240502 14:16:36.475503 28991 consensus_peers.cc:485] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.475628 28991 consensus_peers.cc:485] T 3901ce6537fd48ba861fba13767d8aa4 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.478510 28991 consensus_peers.cc:462] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:36.504302 30990 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:36.504807 31125 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:36.505090 1387 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:36.505299 29314 raft_consensus.cc:1261] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:36.505407 643 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:36.507391 355 consensus_queue.cc:1035] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240502 14:16:36.533300 1387 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:36.534129 31124 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:36.537503 30990 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:36.540046 28907 catalog_manager.cc:5484] T 366dbe98fccd42cb8e369876230984fe P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:36.545307 29471 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session 3bb852e0a21446bbbe824bdefb16036c-63690fff28c84670b15ed02ba1356e33 received from {username='slave'} at 127.21.58.70:56885
I20240502 14:16:36.545460 29471 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session 3bb852e0a21446bbbe824bdefb16036c-63690fff28c84670b15ed02ba1356e33 on tablet 63690fff28c84670b15ed02ba1356e33 with peer 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:36.550805 29386 consensus_peers.cc:485] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.550952 29386 consensus_peers.cc:485] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.551101 29386 consensus_peers.cc:485] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.551227 29386 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:36.551323 29386 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:36.560393 29451 raft_consensus.cc:1203] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:36.560669 29314 raft_consensus.cc:2932] T 366dbe98fccd42cb8e369876230984fe P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:36.595357 29189 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P dbabfd51061b4e509d907a3622febd2e [term 2 FOLLOWER]: Refusing update from remote peer 3bb852e0a21446bbbe824bdefb16036c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:16:36.596051 1680 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:36.606596 30990 raft_consensus.cc:1203] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 14:16:36.618674 29314 raft_consensus.cc:1261] T 44d94461e22c4ae182afa2a15f3db198 P 1b3c20f7bfd548769f6bd40148590b74 [term 2 FOLLOWER]: Refusing update from remote peer 3bb852e0a21446bbbe824bdefb16036c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:16:36.619438 1680 consensus_queue.cc:1035] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:16:36.643337 29432 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:16:36.686021 31125 raft_consensus.cc:1203] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:16:36.707471 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:36.733172 30990 raft_consensus.cc:1203] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 14:16:36.795617 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:36.799786 31107 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:16:36.843475 1563 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:36.843977 1563 tablet_bootstrap.cc:492] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Bootstrap complete.
I20240502 14:16:36.844300 1563 ts_tablet_manager.cc:1367] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Time spent bootstrapping tablet: real 0.996s user 0.062s sys 0.027s
I20240502 14:16:36.858620 29168 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:16:36.879526 1563 raft_consensus.cc:348] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: true } }
I20240502 14:16:36.879969 1563 raft_consensus.cc:729] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Initialized, Role: LEARNER
I20240502 14:16:36.883641 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:36.896449 29451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed"
dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39314
I20240502 14:16:36.896767 29451 raft_consensus.cc:595] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:36.992084 1789 raft_consensus.cc:979] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29: : Instructing follower dbabfd51061b4e509d907a3622febd2e to start an election
I20240502 14:16:36.892416 1563 consensus_queue.cc:260] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: true } }
I20240502 14:16:37.013176 1789 raft_consensus.cc:1067] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Signalling peer dbabfd51061b4e509d907a3622febd2e to start an election
W20240502 14:16:36.991346 31062 connection.cc:535] client connection to 127.21.58.72:36549 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:37.013480 31062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:37.013571 1563 ts_tablet_manager.cc:1398] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e: Time spent starting tablet: real 0.169s user 0.003s sys 0.000s
W20240502 14:16:36.992302 29388 connection.cc:535] client connection to 127.21.58.72:36549 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:37.014412 29189 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed"
dest_uuid: "dbabfd51061b4e509d907a3622febd2e"
from {username='slave'} at 127.21.58.68:45613
I20240502 14:16:37.014672 29189 raft_consensus.cc:482] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:37.014797 29189 raft_consensus.cc:3037] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:37.018110 29189 raft_consensus.cc:504] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:37.018674 29471 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session dbabfd51061b4e509d907a3622febd2e-e6a0a881fcec43b291148f071e4474fc received from {username='slave'} at 127.21.58.66:59129
I20240502 14:16:37.018821 29471 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session dbabfd51061b4e509d907a3622febd2e-e6a0a881fcec43b291148f071e4474fc on tablet e6a0a881fcec43b291148f071e4474fc with peer dbabfd51061b4e509d907a3622febd2e
W20240502 14:16:37.019512 29388 consensus_peers.cc:485] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:37.019687 29388 consensus_peers.cc:485] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.023252 29189 leader_election.cc:288] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 2 election: Requested vote from peers 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70:41819), b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703)
W20240502 14:16:37.022806 31062 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:37.019769 29388 consensus_peers.cc:485] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.042181 31125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3bb852e0a21446bbbe824bdefb16036c"
I20240502 14:16:37.042434 31125 raft_consensus.cc:3037] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:37.045269 31125 raft_consensus.cc:2454] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 2.
I20240502 14:16:37.027478 29451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95b84899c994d45a569d5a4dda124ed" candidate_uuid: "dbabfd51061b4e509d907a3622febd2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "139e612d62744c02bcb5653b7aa1dc29"
I20240502 14:16:37.046685 29451 raft_consensus.cc:3032] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:37.046815 29451 raft_consensus.cc:729] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 139e612d62744c02bcb5653b7aa1dc29, State: Running, Role: LEADER
I20240502 14:16:37.047170 29451 consensus_queue.cc:260] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:37.047683 29451 raft_consensus.cc:3037] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:37.057142 29451 raft_consensus.cc:2454] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbabfd51061b4e509d907a3622febd2e in term 2.
W20240502 14:16:36.900570 28991 consensus_peers.cc:462] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:36.957401 1544 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 14:16:37.080595 28991 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.21.58.71:44703, user_credentials={real_user=slave}} blocked reactor thread for 180018us
I20240502 14:16:37.081126 1544 tablet_bootstrap.cc:492] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Bootstrap complete.
W20240502 14:16:36.988708 29257 connection.cc:535] client connection to 127.21.58.72:36549 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:37.081444 1544 ts_tablet_manager.cc:1367] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Time spent bootstrapping tablet: real 1.626s user 0.058s sys 0.007s
I20240502 14:16:37.082259 1544 raft_consensus.cc:348] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } }
I20240502 14:16:37.082517 1544 raft_consensus.cc:729] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9329e8a7a2d1453d90b76b5965315837, State: Initialized, Role: LEARNER
I20240502 14:16:37.082715 1544 consensus_queue.cc:260] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } }
W20240502 14:16:36.992568 28993 connection.cc:535] client connection to 127.21.58.72:36549 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:37.083108 28991 consensus_peers.cc:462] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:37.040007 1750 raft_consensus.cc:1050] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER 139e612d62744c02bcb5653b7aa1dc29 to VOTER
I20240502 14:16:37.105984 643 raft_consensus.cc:1050] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
W20240502 14:16:37.107920 29257 consensus_peers.cc:485] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.134227 31439 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3cgK4A/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:16:18.423302 31445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:18.423754 31445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:18.437341 31445 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:21.432083 31445 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:16:23.019343 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.019527 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:23.019681 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.019886 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:23.019999 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.020134 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.020217 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.020346 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.020439 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.020591 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:23.020679 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.020792 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:23.020912 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.021013 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.021083 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server b7dacf388234446b9a11ce750978ca75
I20240502 14:16:23.171885 31445 rebalancer_tool.cc:1074] tablet e6a0a881fcec43b291148f071e4474fc: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:23.416244 31445 rebalancer_tool.cc:1074] tablet a6eb9dee735a4115874ef93636a559ab: '1b3c20f7bfd548769f6bd40148590b74' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:23.652215 31445 rebalancer_tool.cc:1074] tablet 432dc9a199294b748414520a2c49a674: '9329e8a7a2d1453d90b76b5965315837' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:23.848675 31445 rebalancer_tool.cc:1074] tablet 44d94461e22c4ae182afa2a15f3db198: '139e612d62744c02bcb5653b7aa1dc29' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:24.135388 31445 rebalancer_tool.cc:1074] tablet 96fb0fc5919d42d7bedd770945935b42: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:24.304777 31445 rebalancer_tool.cc:1074] tablet a0fcde0e7cbe41f1a521e669b798aa99: '1b3c20f7bfd548769f6bd40148590b74' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:24.504062 31445 rebalancer_tool.cc:1074] tablet fd4e4241d42d483ab6203606a4d92141: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:24.744477 31445 rebalancer_tool.cc:1074] tablet d67ee220c86f4fc7bc1db8ae08d668d1: '139e612d62744c02bcb5653b7aa1dc29' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:25.318589 31445 rebalancer_tool.cc:1084] tablet 981ec6067d624100bddba4383b417c50: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: NON_VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false }W20240502 14:16:37.108075 29257 consensus_peers.cc:485] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:37.229513 29257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.58.72:36549, user_credentials={real_user=slave}} blocked reactor thread for 121376us
I20240502 14:16:37.107010 643 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
W20240502 14:16:37.229705 29257 consensus_peers.cc:462] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
I20240502 14:16:37.186924 1387 raft_consensus.cc:1050] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER 3bb852e0a21446bbbe824bdefb16036c to VOTER
I20240502 14:16:37.232265 1387 consensus_queue.cc:237] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:37.236114 30990 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:37.236649 31125 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:37.236819 1387 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
}
I20240502 14:16:25.318748 31445 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:28.203680 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:28.203891 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:28.204033 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:28.204177 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:28.204447 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:28.205015 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:28.205130 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:28.205216 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:28.205559 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:28.206018 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:28.206135 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:28.206293 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:28.206718 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:28.207175 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:28.498798 31445 rebalancer_tool.cc:1074] tablet 981ec6067d624100bddba4383b417c50: 'dbabfd51061b4e509d907a3622febd2e' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:28.734536 31445 rebalancer_tool.cc:1074] tablet d95b84899c994d45a569d5a4dda124ed: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:29.049513 31445 rebalancer_tool.cc:1074] tablet 239ee601d297412494cf2b447ef75e9a: '1b3c20f7bfd548769f6bd40148590b74' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:29.684070 31445 rebalancer_tool.cc:1074] tablet 0272c80ea41149f895074e0e16180b22: '139e612d62744c02bcb5653b7aa1dc29' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:30.028142 31445 rebalancer_tool.cc:1074] tablet e89c7a017200449abcd5d9b9c2bcc912: 'dbabfd51061b4e509d907a3622febd2e' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:30.246702 31445 rebalancer_tool.cc:1074] tablet a00d9c46bfc64a4085715e4f764b8c96: '139e612d62744c02bcb5653b7aa1dc29' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:30.574736 31445 rebalancer_tool.cc:1074] tablet 093e29d30b2646d68e84b8b9ca86ca9e: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:30.923367 31445 rebalancer_tool.cc:1074] tablet 1f17889f58a34bc0afa40a246e5debc6: '1b3c20f7bfd548769f6bd40148590b74' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:31.300405 31445 rebalancer_tool.cc:1084] tablet c4818cd7d9b44111a04a0d32a4580cdf: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move ignored: Illegal state: tablet c4818cd7d9b44111a04a0d32a4580cdf: replica b7dacf388234446b9a11ce750978ca75 is already present
I20240502 14:16:31.300578 31445 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:33.944268 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:33.944418 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server 139e612d62744c02bcb5653b7aa1dc29
W20240502 14:16:33.945084 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:33.945703 31445 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:34.202849 31445 rebalancer_tool.cc:1074] tablet 302978e1f6034b3693b3a0abacdc494e: 'b7dacf388234446b9a11ce750978ca75' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:34.491919 31445 rebalancer_tool.cc:1074] tablet f4dbd9bc62484dd1998080d289db098f: '139e612d62744c02bcb5653b7aa1dc29' -> '9329e8a7a2d1453d90b76b5965315837' move scheduled
I20240502 14:16:34.574280 31445 rebalancer_tool.cc:1084] tablet a230503874a348f4a19647179fe25123: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move ignored: Illegal state: tablet a230503874a348f4a19647179fe25123: replica b7dacf388234446b9a11ce750978ca75 is already present
I20240502 14:16:34.574402 31445 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:36.348129 1661 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:35.583476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.58.71:44703 (local address 127.0.0.1:46422)
0502 14:16:35.599507 (+ 16031us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:35.599522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 14:16:35.599618 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:16:36.346638 (+747020us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15930,"thread_start_us":30353,"threads_started":1}
Illegal state: tablet server b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): unacceptable health status UNAVAILABLE
I20240502 14:16:37.253393 31125 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:37.238201 1878 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:16:37.254691 29386 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.254135 30990 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 14:16:37.029397 29123 leader_election.cc:334] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
W20240502 14:16:37.288911 29123 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.58.71:44703, user_credentials={real_user=slave}} blocked reactor thread for 259471us
I20240502 14:16:37.086197 1750 consensus_queue.cc:237] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } }
I20240502 14:16:37.119446 1544 ts_tablet_manager.cc:1398] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20240502 14:16:37.289875 1387 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:16:37.146559 28993 consensus_peers.cc:462] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
W20240502 14:16:37.290271 28993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.21.58.72:36549, user_credentials={real_user=slave}} blocked reactor thread for 143740us
W20240502 14:16:37.290431 28993 consensus_peers.cc:485] T 093e29d30b2646d68e84b8b9ca86ca9e P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:37.290563 28993 consensus_peers.cc:485] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:37.290675 28993 consensus_peers.cc:485] T 3fde0741532a4067a4806f4bc0b0d465 P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.223548 31438 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3cgK4A/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:16:18.251698 31446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:18.252161 31446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:18.267772 31446 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:21.399920 31446 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:16:23.201000 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.201668 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.201858 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.201999 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.202234 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.205507 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.205670 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.206461 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.206588 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.206807 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:23.206949 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.207130 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.207243 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.207432 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.207589 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:23.207772 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:23.207943 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:23.208082 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:23.519999 31446 rebalancer_tool.cc:1084] tablet e6a0a881fcec43b291148f071e4474fc: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move ignored: Illegal state: tablet e6a0a881fcec43b291148f071e4474fc: replica b7dacf388234446b9a11ce750978ca75 is already present
I20240502 14:16:23.520191 31446 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:25.615391 32294 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.21.58.68:46509, user_credentials={real_user=slave}} blocked reactor thread for 85001.8us
W20240502 14:16:25.894937 32294 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.21.58.65:46783, user_credentials={real_user=slave}} blocked reactor thread for 135156us
W20240502 14:16:27.094566 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:27.094794 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:27.095153 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:27.095275 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:27.095363 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:27.095512 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:27.095706 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:27.095839 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:27.157063 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:27.157338 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:27.157997 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:27.158154 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:27.158248 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:27.158370 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:27.158457 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:27.158569 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:27.158656 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:27.158775 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:27.158882 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:27.158984 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:27.159058 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:27.192123 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:27.192248 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:27.193284 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:27.843854 31446 rebalancer_tool.cc:1074] tablet 71f047f74165489bacb4f72ab981b447: '9329e8a7a2d1453d90b76b5965315837' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:28.402838 31446 rebalancer_tool.cc:1074] tablet c4818cd7d9b44111a04a0d32a4580cdf: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:28.754038 31446 rebalancer_tool.cc:1084] tablet 39835ac7b2bb4c0a8c4b43b010b09706: '139e612d62744c02bcb5653b7aa1dc29' -> 'ede5e1f11f254824a335e1e43f40b6d1' move ignored: Illegal state: tablet 39835ac7b2bb4c0a8c4b43b010b09706: replica ede5e1f11f254824a335e1e43f40b6d1 is already present
I20240502 14:16:28.754218 31446 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:31.589262 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:31.949232 31446 rebalancer_tool.cc:1074] tablet 44d94461e22c4ae182afa2a15f3db198: 'dbabfd51061b4e509d907a3622febd2e' -> '' move scheduled
I20240502 14:16:32.200284 31446 rebalancer_tool.cc:1084] tablet e6a0a881fcec43b291148f071e4474fc: '1b3c20f7bfd548769f6bd40148590b74' -> 'ede5e1f11f254824a335e1e43f40b6d1' move ignored: Illegal state: tablet e6a0a881fcec43b291148f071e4474fc: replica ede5e1f11f254824a335e1e43f40b6d1 is already present
I20240502 14:16:32.200459 31446 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:34.171995 31446 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:34.355603 31446 rebalancer_tool.cc:1084] tablet 302978e1f6034b3693b3a0abacdc494e: 'b7dacf388234446b9a11ce750978ca75' -> 'ede5e1f11f254824a335e1e43f40b6d1' move ignored: Illegal state: tablet 302978e1f6034b3693b3a0abacdc494e: replica ede5e1f11f254824a335e1e43f40b6d1 is already present
I20240502 14:16:34.355757 31446 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:36.348086 1636 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:35.619897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.58.71:44703 (local address 127.0.0.1:46432)
0502 14:16:35.620062 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:35.620082 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0502 14:16:35.620176 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:16:36.346635 (+726459us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
Illegal state: tablet server b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): unacceptable health status UNAVAILABLE
I20240502 14:16:37.051541 29189 raft_consensus.cc:1203] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
W20240502 14:16:37.292194 29124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.58.70:41819, user_credentials={real_user=slave}} blocked reactor thread for 246367us
I20240502 14:16:37.147727 1854 raft_consensus.cc:1050] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837: attempting to promote NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 to VOTER
I20240502 14:16:37.292697 29124 leader_election.cc:302] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 139e612d62744c02bcb5653b7aa1dc29, 3bb852e0a21446bbbe824bdefb16036c, dbabfd51061b4e509d907a3622febd2e; no voters: b7dacf388234446b9a11ce750978ca75
I20240502 14:16:37.293417 1854 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } }
I20240502 14:16:37.294497 1740 raft_consensus.cc:2781] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:37.294641 1740 raft_consensus.cc:686] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [term 2 LEADER]: Becoming Leader. State: Replica: dbabfd51061b4e509d907a3622febd2e, State: Running, Role: LEADER
I20240502 14:16:37.294911 1740 consensus_queue.cc:237] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:37.294929 1387 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:37.298014 29471 tablet_copy_service.cc:342] P 139e612d62744c02bcb5653b7aa1dc29: Request end of tablet copy session 9329e8a7a2d1453d90b76b5965315837-f4dbd9bc62484dd1998080d289db098f received from {username='slave'} at 127.21.58.65:47501
I20240502 14:16:37.298168 29471 tablet_copy_service.cc:433] P 139e612d62744c02bcb5653b7aa1dc29: ending tablet copy session 9329e8a7a2d1453d90b76b5965315837-f4dbd9bc62484dd1998080d289db098f on tablet f4dbd9bc62484dd1998080d289db098f with peer 9329e8a7a2d1453d90b76b5965315837
I20240502 14:16:37.235847 29185 raft_consensus.cc:1261] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:16:37.300452 28993 consensus_peers.cc:462] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
I20240502 14:16:37.300838 355 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:37.303686 31125 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:37.310763 355 consensus_queue.cc:1035] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:37.310657 30992 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
W20240502 14:16:37.313849 28993 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.315021 30989 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:37.315708 1415 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:37.316970 28907 catalog_manager.cc:5484] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 reported cstate change: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:16:37.321238 28991 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:37.322247 29257 consensus_peers.cc:462] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
I20240502 14:16:37.322428 29314 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:37.323052 1415 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:16:37.341095 29185 raft_consensus.cc:2932] T 63690fff28c84670b15ed02ba1356e33 P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3bb852e0a21446bbbe824bdefb16036c (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } }
I20240502 14:16:37.353296 1415 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:37.365796 28915 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:37.370894 29314 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:37.375707 30989 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: NON_VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: true } } }
I20240502 14:16:37.388391 29056 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
W20240502 14:16:37.392189 29124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.58.68:46509, user_credentials={real_user=slave}} blocked reactor thread for 99476.3us
W20240502 14:16:37.394651 28991 consensus_peers.cc:485] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.398149 30992 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:37.398308 28915 catalog_manager.cc:5484] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e reported cstate change: term changed from 1 to 2, leader changed from 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) to dbabfd51061b4e509d907a3622febd2e (127.21.58.66). New cstate: current_term: 2 leader_uuid: "dbabfd51061b4e509d907a3622febd2e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:37.398743 29315 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:37.400048 1415 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:37.400311 1855 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:37.406749 29451 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEARNER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:37.407346 1855 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:16:37.407027 1854 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.408553 1387 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.411650 29316 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.412442 30990 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.415179 1881 raft_consensus.cc:1050] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER dbabfd51061b4e509d907a3622febd2e to VOTER
I20240502 14:16:37.415745 29315 raft_consensus.cc:1261] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:16:37.416031 1881 consensus_queue.cc:237] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: false } }
I20240502 14:16:37.416688 1856 consensus_queue.cc:1035] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:16:37.419093 28993 consensus_peers.cc:485] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.419903 28915 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 11 to 12, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.424492 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96fb0fc5919d42d7bedd770945935b42 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:37.425333 30990 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.431505 31125 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.433174 28909 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 reported cstate change: config changed from index 10 to 11, ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.445766 29056 consensus_queue.cc:237] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
W20240502 14:16:37.448879 28991 consensus_peers.cc:462] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:37.449479 30990 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:16:37.450098 1856 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:16:37.452265 29314 raft_consensus.cc:1261] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 9329e8a7a2d1453d90b76b5965315837: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:16:37.453070 1856 consensus_queue.cc:1035] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:16:37.454569 1415 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.456317 30989 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.457752 29314 raft_consensus.cc:2932] T 96fb0fc5919d42d7bedd770945935b42 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.458642 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96fb0fc5919d42d7bedd770945935b42 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:37.462661 28907 catalog_manager.cc:5484] T 96fb0fc5919d42d7bedd770945935b42 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 12 to 13, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:37.472138 29452 consensus_queue.cc:237] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } }
I20240502 14:16:37.477679 30989 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:37.478160 31125 raft_consensus.cc:1261] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:37.479102 1387 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:37.479676 1387 consensus_queue.cc:1035] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:37.483543 643 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.488008 31125 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.491385 1854 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:37.496209 29451 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 139e612d62744c02bcb5653b7aa1dc29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
W20240502 14:16:37.497578 28900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:37.497619 29056 raft_consensus.cc:1203] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 14:16:37.501806 28991 consensus_peers.cc:462] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:37.506695 30989 raft_consensus.cc:2932] T 302978e1f6034b3693b3a0abacdc494e P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.507877 28909 catalog_manager.cc:5484] T 302978e1f6034b3693b3a0abacdc494e P 3bb852e0a21446bbbe824bdefb16036c reported cstate change: config changed from index 11 to 12, VOTER b7dacf388234446b9a11ce750978ca75 (127.21.58.71) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } }
I20240502 14:16:37.510005 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 302978e1f6034b3693b3a0abacdc494e with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:37.510918 29314 raft_consensus.cc:2932] T 0272c80ea41149f895074e0e16180b22 P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } } }
I20240502 14:16:37.516937 28907 catalog_manager.cc:5484] T 0272c80ea41149f895074e0e16180b22 P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 7 to 8, 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9329e8a7a2d1453d90b76b5965315837" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:16:37.531170 28900 catalog_manager.cc:4573] TS b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 302978e1f6034b3693b3a0abacdc494e: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:37.540531 29189 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:16:37.541186 1387 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:16:37.542726 30989 raft_consensus.cc:1261] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:16:37.543356 1387 consensus_queue.cc:1035] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 14:16:37.553623 29386 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.557374 1881 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, dbabfd51061b4e509d907a3622febd2e (127.21.58.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: false } } }
I20240502 14:16:37.563035 29189 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, dbabfd51061b4e509d907a3622febd2e (127.21.58.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: false } } }
I20240502 14:16:37.564086 30989 raft_consensus.cc:2932] T e6a0a881fcec43b291148f071e4474fc P ede5e1f11f254824a335e1e43f40b6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, dbabfd51061b4e509d907a3622febd2e (127.21.58.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: false } } }
I20240502 14:16:37.570012 28907 catalog_manager.cc:5484] T e6a0a881fcec43b291148f071e4474fc P dbabfd51061b4e509d907a3622febd2e reported cstate change: config changed from index 12 to 13, dbabfd51061b4e509d907a3622febd2e (127.21.58.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } attrs { promote: false } } }
I20240502 14:16:37.615278 31441 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3cgK4A/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:16:18.593343 31447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:18.593827 31447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:18.608891 31447 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:22.184077 31447 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:16:24.418902 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:24.419211 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:24.419289 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:24.425282 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:24.425467 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.425571 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.425650 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:24.430115 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:24.430236 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.430327 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:24.430439 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:24.430546 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.430717 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.431448 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:24.431552 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:24.432363 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:24.432472 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:24.554355 31447 rebalancer_tool.cc:1074] tablet fd4e4241d42d483ab6203606a4d92141: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:24.686720 31447 rebalancer_tool.cc:1074] tablet f4dbd9bc62484dd1998080d289db098f: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:25.325315 31447 rebalancer_tool.cc:1074] tablet e66f9bb135bf4007b1864b63a389fac3: '139e612d62744c02bcb5653b7aa1dc29' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:26.186300 31447 rebalancer_tool.cc:1074] tablet a0fcde0e7cbe41f1a521e669b798aa99: '1b3c20f7bfd548769f6bd40148590b74' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:26.582077 31447 rebalancer_tool.cc:1074] tablet 302978e1f6034b3693b3a0abacdc494e: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:27.007630 31447 rebalancer_tool.cc:1074] tablet 366dbe98fccd42cb8e369876230984fe: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:27.349807 31447 rebalancer_tool.cc:1084] tablet e89c7a017200449abcd5d9b9c2bcc912: '139e612d62744c02bcb5653b7aa1dc29' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move ignored: Not found: Peer with uuid 139e612d62744c02bcb5653b7aa1dc29 not found in consensus config
I20240502 14:16:27.349946 31447 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:29.962478 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:29.964469 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:29.965124 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:29.965415 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:29.965690 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:29.966077 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:29.966395 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:29.966590 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:30.127876 31447 rebalancer_tool.cc:1074] tablet 44d94461e22c4ae182afa2a15f3db198: 'dbabfd51061b4e509d907a3622febd2e' -> '' move scheduled
I20240502 14:16:30.403486 31447 rebalancer_tool.cc:1074] tablet 3fde0741532a4067a4806f4bc0b0d465: '139e612d62744c02bcb5653b7aa1dc29' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:30.939369 31447 rebalancer_tool.cc:1074] tablet d67ee220c86f4fc7bc1db8ae08d668d1: '1b3c20f7bfd548769f6bd40148590b74' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:31.201555 31447 rebalancer_tool.cc:1074] tablet 3901ce6537fd48ba861fba13767d8aa4: '9329e8a7a2d1453d90b76b5965315837' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:31.520349 31447 rebalancer_tool.cc:1084] tablet e89c7a017200449abcd5d9b9c2bcc912: '1b3c20f7bfd548769f6bd40148590b74' -> '3bb852e0a21446bbbe824bdefb16036c' move ignored: Illegal state: tablet e89c7a017200449abcd5d9b9c2bcc912: replica 3bb852e0a21446bbbe824bdefb16036c is already present
I20240502 14:16:31.520469 31447 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:33.845443 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 9329e8a7a2d1453d90b76b5965315837
W20240502 14:16:33.875383 31447 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:34.097559 31447 rebalancer_tool.cc:1074] tablet 981ec6067d624100bddba4383b417c50: '9329e8a7a2d1453d90b76b5965315837' -> '' move scheduled
I20240502 14:16:34.311254 31447 rebalancer_tool.cc:1074] tablet 96fb0fc5919d42d7bedd770945935b42: 'b7dacf388234446b9a11ce750978ca75' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:34.649818 31447 rebalancer_tool.cc:1074] tablet 63690fff28c84670b15ed02ba1356e33: '139e612d62744c02bcb5653b7aa1dc29' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:34.767977 31447 rebalancer_tool.cc:1074] tablet e6a0a881fcec43b291148f071e4474fc: '139e612d62744c02bcb5653b7aa1dc29' -> 'dbabfd51061b4e509d907a3622febd2e' move scheduled
I20240502 14:16:35.173167 31447 rebalancer_tool.cc:1074] tablet a00d9c46bfc64a4085715e4f764b8c96: '9329e8a7a2d1453d90b76b5965315837' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:35.498288 31447 rebalancer_tool.cc:1084] tablet a230503874a348f4a19647179fe25123: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move ignored: Illegal state: tablet a230503874a348f4a19647179fe25123: replica b7dacf388234446b9a11ce750978ca75 is already present
I20240502 14:16:35.498479 31447 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:36.548768 1699 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.21.58.126:42921, user_credentials={real_user=slave}} blocked reactor thread for 62293.4us
W20240502 14:16:36.994637 1791 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:36.638286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.58.72:36549 (local address 127.0.0.1:47816)
0502 14:16:36.667310 (+ 29024us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:36.667329 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0502 14:16:36.676720 (+ 9391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:16:36.993163 (+316443us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26584}
Illegal state: tablet server b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): unacceptable health status UNAVAILABLE
I20240502 14:16:37.619189 29452 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:37.621942 29056 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:37.622502 29314 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:37.622844 1878 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:37.623127 31125 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:16:37.623695 1878 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:16:37.624135 29385 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4dbd9bc62484dd1998080d289db098f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.624648 1878 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:37.629058 1878 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.632434 29314 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.638674 28915 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 reported cstate change: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.639588 1387 raft_consensus.cc:1050] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29: attempting to promote NON_VOTER 9329e8a7a2d1453d90b76b5965315837 to VOTER
I20240502 14:16:37.640347 1387 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:37.642608 29314 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:37.643173 355 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:16:37.643482 29385 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4dbd9bc62484dd1998080d289db098f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.645542 31124 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:37.646304 355 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:37.646481 29057 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEARNER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:16:37.648439 31124 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.649433 355 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:16:37.651854 29057 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.660532 355 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.661299 29057 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.664729 31125 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.666630 29314 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.667454 28909 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 10 to 11, 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } attrs { replace: true } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.681878 29452 consensus_queue.cc:237] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } }
I20240502 14:16:37.684258 29057 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 14:16:37.684417 29385 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4dbd9bc62484dd1998080d289db098f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.684782 1881 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:37.686476 31124 raft_consensus.cc:1261] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Refusing update from remote peer 139e612d62744c02bcb5653b7aa1dc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:16:37.687254 355 consensus_queue.cc:1035] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:16:37.690137 1878 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.690649 29057 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.691790 31125 raft_consensus.cc:2932] T f4dbd9bc62484dd1998080d289db098f P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.693672 28901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4dbd9bc62484dd1998080d289db098f with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:16:37.695549 28909 catalog_manager.cc:5484] T f4dbd9bc62484dd1998080d289db098f P 9329e8a7a2d1453d90b76b5965315837 reported cstate change: config changed from index 11 to 12, VOTER 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "139e612d62744c02bcb5653b7aa1dc29" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { promote: false } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: true } } }
I20240502 14:16:37.698788 29301 tablet_service.cc:1514] Processing DeleteTablet for tablet f4dbd9bc62484dd1998080d289db098f with delete_type TABLET_DATA_TOMBSTONED (TS 1b3c20f7bfd548769f6bd40148590b74 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:57716
I20240502 14:16:37.699415 1946 tablet_replica.cc:330] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: stopping tablet replica
I20240502 14:16:37.699956 1946 raft_consensus.cc:2229] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:37.700345 1946 raft_consensus.cc:2258] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:37.701386 1946 ts_tablet_manager.cc:1875] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:16:37.704052 29257 consensus_peers.cc:462] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
I20240502 14:16:37.711166 1946 ts_tablet_manager.cc:1888] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 14:16:37.711283 1946 log.cc:1196] T f4dbd9bc62484dd1998080d289db098f P 1b3c20f7bfd548769f6bd40148590b74: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4dbd9bc62484dd1998080d289db098f
I20240502 14:16:37.712113 28899 catalog_manager.cc:4830] TS 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): tablet f4dbd9bc62484dd1998080d289db098f (table rebalance_test_table_1 [id=9c72cf765d124fa69d0f3f17b1247b14]) successfully deleted
W20240502 14:16:37.743803 29123 consensus_peers.cc:485] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:37.765981 29452 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 139e612d62744c02bcb5653b7aa1dc29 [term 2 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:16:37.767043 1886 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:16:37.856536 28993 consensus_peers.cc:462] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
I20240502 14:16:37.886664 31125 raft_consensus.cc:1261] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 2 FOLLOWER]: Refusing update from remote peer dbabfd51061b4e509d907a3622febd2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:16:37.887483 1886 consensus_queue.cc:1035] T d95b84899c994d45a569d5a4dda124ed P dbabfd51061b4e509d907a3622febd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:16:37.973691 28991 consensus_peers.cc:462] T a230503874a348f4a19647179fe25123 P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:37.979275 29432 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:38.017465 29432 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:38.019204 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:16:38.040520 28991 consensus_peers.cc:462] T b7bfc3bb327b4407ae1aa7742f4b18de P 9329e8a7a2d1453d90b76b5965315837 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111)
I20240502 14:16:38.046816 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:38.058229 29301 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:38.060999 31107 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:16:38.075186 31107 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:16:38.093009 29037 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:16:38.096130 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:38.150502 29168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:16:38.157258 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:38.187980 30972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:38.223460 2028 ts_tablet_manager.cc:902] T f4dbd9bc62484dd1998080d289db098f P ede5e1f11f254824a335e1e43f40b6d1: Initiating tablet copy from peer 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509)
F20240502 14:16:38.223984 2028 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.queue_time_us, tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.queue_time_us) (tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_cpu_time_us, tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_cpu_time_us) (tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_wall_time_us, tablet-download-pool-1f17889f58a34bc0afa40a246e5debc6.run_wall_time_us) (tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.queue_time_us, tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.queue_time_us) (tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_cpu_time_us, tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_cpu_time_us) (tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_wall_time_us, tablet-download-pool-302978e1f6034b3693b3a0abacdc494e.run_wall_time_us) (tablet-download-pool-366dbe98fccd42cb8e369876230984fe.queue_time_us, tablet-download-pool-366dbe98fccd42cb8e369876230984fe.queue_time_us) (tablet-download-pool-366dbe98fccd42cb8e369876230984fe.run_cpu_time_us, tablet-download-pool-366dbe98fccd42cb8e369876230984fe.run_cpu_time_us) (tablet-download-pool-366dbe98fccd42cb8e369876230984fe.run_wall_time_us, tablet-download-pool-366dbe98fccd42cb8e369876230984fe.run_wall_time_us) (tablet-download-pool-39835ac7b2bb4c0a8c4b43b010b09706.queue_time_us, tablet-download-pool-39835ac7b2bb4c0a8c4b43b010b09706.queue_time_us) (tablet-download-pool-39835ac7b2bb4c0a8c4b43b010b09706.run_cpu_time_us, tablet-download-pool-39835ac7b2bb4c0a8c4b43b010b09706.run_cpu_time_us) (tablet-download-pool-39835ac7b2bb4c0a8c4b43b010b09706.run_wall_time_us, tablet-download-pool-39835ac7b2bb4c0a8c4b43b010b09706.run_wall_time_us) (tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.queue_time_us, tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.queue_time_us) (tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_cpu_time_us, tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_cpu_time_us) (tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_wall_time_us, tablet-download-pool-3fde0741532a4067a4806f4bc0b0d465.run_wall_time_us) (tablet-download-pool-432dc9a199294b748414520a2c49a674.queue_time_us, tablet-download-pool-432dc9a199294b748414520a2c49a674.queue_time_us) (tablet-download-pool-432dc9a199294b748414520a2c49a674.run_cpu_time_us, tablet-download-pool-432dc9a199294b748414520a2c49a674.run_cpu_time_us) (tablet-download-pool-432dc9a199294b748414520a2c49a674.run_wall_time_us, tablet-download-pool-432dc9a199294b748414520a2c49a674.run_wall_time_us) (tablet-download-pool-63690fff28c84670b15ed02ba1356e33.queue_time_us, tablet-download-pool-63690fff28c84670b15ed02ba1356e33.queue_time_us) (tablet-download-pool-63690fff28c84670b15ed02ba1356e33.run_cpu_time_us, tablet-download-pool-63690fff28c84670b15ed02ba1356e33.run_cpu_time_us) (tablet-download-pool-63690fff28c84670b15ed02ba1356e33.run_wall_time_us, tablet-download-pool-63690fff28c84670b15ed02ba1356e33.run_wall_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.queue_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.queue_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_cpu_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_cpu_time_us) (tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_wall_time_us, tablet-download-pool-96fb0fc5919d42d7bedd770945935b42.run_wall_time_us) (tablet-download-pool-981ec6067d624100bddba4383b417c50.queue_time_us, tablet-download-pool-981ec6067d624100bddba4383b417c50.queue_time_us) (tablet-download-pool-981ec6067d624100bddba4383b417c50.run_cpu_time_us, tablet-download-pool-981ec6067d624100bddba4383b417c50.run_cpu_time_us) (tablet-download-pool-981ec6067d624100bddba4383b417c50.run_wall_time_us, tablet-download-pool-981ec6067d624100bddba4383b417c50.run_wall_time_us) (tablet-download-pool-a00d9c46bfc64a4085715e4f764b8c96.queue_time_us, tablet-download-pool-a00d9c46bfc64a4085715e4f764b8c96.queue_time_us) (tablet-download-pool-a00d9c46bfc64a4085715e4f764b8c96.run_cpu_time_us, tablet-download-pool-a00d9c46bfc64a4085715e4f764b8c96.run_cpu_time_us) (tablet-download-pool-a00d9c46bfc64a4085715e4f764b8c96.run_wall_time_us, tablet-download-pool-a00d9c46bfc64a4085715e4f764b8c96.run_wall_time_us) (tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.queue_time_us, tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.queue_time_us) (tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_cpu_time_us, tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_cpu_time_us) (tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_wall_time_us, tablet-download-pool-a0fcde0e7cbe41f1a521e669b798aa99.run_wall_time_us) (tablet-download-pool-a230503874a348f4a19647179fe25123.queue_time_us, tablet-download-pool-a230503874a348f4a19647179fe25123.queue_time_us) (tablet-download-pool-a230503874a348f4a19647179fe25123.run_cpu_time_us, tablet-download-pool-a230503874a348f4a19647179fe25123.run_cpu_time_us) (tablet-download-pool-a230503874a348f4a19647179fe25123.run_wall_time_us, tablet-download-pool-a230503874a348f4a19647179fe25123.run_wall_time_us) (tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.queue_time_us, tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.queue_time_us) (tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_cpu_time_us, tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_cpu_time_us) (tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_wall_time_us, tablet-download-pool-d67ee220c86f4fc7bc1db8ae08d668d1.run_wall_time_us) (tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.queue_time_us, tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.queue_time_us) (tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_cpu_time_us, tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_cpu_time_us) (tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_wall_time_us, tablet-download-pool-e6a0a881fcec43b291148f071e4474fc.run_wall_time_us) (tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.queue_time_us, tablet-download-pool-f4dbd9bc62484dd1998080d289db098f.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f6b38a30dcd google::LogMessage::Fail() at ??:0
@ 0x7f6b38a34b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6b38a307cc google::LogMessage::Flush() at ??:0
@ 0x7f6b38a31f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6b3b771742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f6b3b73df8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6b3b737cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f6b4ab8a8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f6b4aba2daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f6b4accd480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f6b4ad05a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f6b4ace8981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f6b4ace870e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240502 14:16:38.287901 29257 consensus_peers.cc:462] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
@ 0x7f6b3b741b75 kudu::ThreadPool::DispatchThread() at ??:0
W20240502 14:16:38.293524 28993 consensus_peers.cc:462] T 239ee601d297412494cf2b447ef75e9a P 9329e8a7a2d1453d90b76b5965315837 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
@ 0x7f6b3b74519c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6b3b745019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f6b3b72001e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6b3eaa56db start_thread at ??:0
@ 0x7f6b378e371f clone at ??:0
W20240502 14:16:38.433691 29255 consensus_peers.cc:485] T 078ffca14316466aa3906d2ff0567c70 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:38.500159 31440 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3cgK4A/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:16:18.701453 31448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:18.703698 31448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:18.733912 31448 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 14:16:19.970059 31666 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.21.58.126:42921, user_credentials={real_user=slave}} blocked reactor thread for 84743.5us
I20240502 14:16:22.473459 31448 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:16:24.434680 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:24.455261 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.455456 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:24.455577 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.455735 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:24.455936 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:24.456049 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:24.456157 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:24.456326 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.456460 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:24.456563 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.456657 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:24.456772 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:24.456854 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:24.457010 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:24.462184 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
I20240502 14:16:25.071781 31448 rebalancer_tool.cc:1074] tablet e6a0a881fcec43b291148f071e4474fc: 'dbabfd51061b4e509d907a3622febd2e' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:26.011153 31448 rebalancer_tool.cc:1074] tablet f4dbd9bc62484dd1998080d289db098f: '9329e8a7a2d1453d90b76b5965315837' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:26.408658 31448 rebalancer_tool.cc:1074] tablet 71f047f74165489bacb4f72ab981b447: '1b3c20f7bfd548769f6bd40148590b74' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:26.796278 31448 rebalancer_tool.cc:1074] tablet 39835ac7b2bb4c0a8c4b43b010b09706: '139e612d62744c02bcb5653b7aa1dc29' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:27.529050 31448 rebalancer_tool.cc:1074] tablet d3221d1a1b244a27abc9ee26e0a6edb1: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:27.827689 31448 rebalancer_tool.cc:1084] tablet 1f17889f58a34bc0afa40a246e5debc6: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move ignored: Not found: Peer with uuid 9329e8a7a2d1453d90b76b5965315837 not found in consensus config
I20240502 14:16:27.827931 31448 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:30.331122 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:30.334573 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:30.334810 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:30.334997 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
I20240502 14:16:30.627843 31448 rebalancer_tool.cc:1074] tablet fd4e4241d42d483ab6203606a4d92141: '1b3c20f7bfd548769f6bd40148590b74' -> '' move scheduled
I20240502 14:16:30.883822 31448 rebalancer_tool.cc:1074] tablet 432dc9a199294b748414520a2c49a674: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:31.145707 31448 rebalancer_tool.cc:1084] tablet a00d9c46bfc64a4085715e4f764b8c96: '9329e8a7a2d1453d90b76b5965315837' -> 'ede5e1f11f254824a335e1e43f40b6d1' move ignored: Illegal state: tablet a00d9c46bfc64a4085715e4f764b8c96: replica ede5e1f11f254824a335e1e43f40b6d1 is already present
I20240502 14:16:31.145856 31448 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:33.648826 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server d653f68cb53b4ed79d5e688987ed8d4c to server 9329e8a7a2d1453d90b76b5965315837
W20240502 14:16:33.649022 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:33.649164 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:33.649458 31448 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server d653f68cb53b4ed79d5e688987ed8d4c to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:33.894053 31448 rebalancer_tool.cc:1074] tablet 96fb0fc5919d42d7bedd770945935b42: 'b7dacf388234446b9a11ce750978ca75' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:33.986858 31448 rebalancer_tool.cc:1074] tablet 44d94461e22c4ae182afa2a15f3db198: 'dbabfd51061b4e509d907a3622febd2e' -> '' move scheduled
I20240502 14:16:34.276377 31448 rebalancer_tool.cc:1074] tablet a230503874a348f4a19647179fe25123: '9329e8a7a2d1453d90b76b5965315837' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:34.478317 31448 rebalancer_tool.cc:1074] tablet 0272c80ea41149f895074e0e16180b22: '1b3c20f7bfd548769f6bd40148590b74' -> '139e612d62744c02bcb5653b7aa1dc29' move scheduled
I20240502 14:16:34.603688 31448 rebalancer_tool.cc:1074] tablet 239ee601d297412494cf2b447ef75e9a: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:34.858659 31448 rebalancer_tool.cc:1074] tablet 366dbe98fccd42cb8e369876230984fe: '139e612d62744c02bcb5653b7aa1dc29' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:35.107000 31448 rebalancer_tool.cc:1074] tablet b7bfc3bb327b4407ae1aa7742f4b18de: '9329e8a7a2d1453d90b76b5965315837' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:36.912066 31448 rebalancer_tool.cc:1136] tablet 432dc9a199294b748414520a2c49a674: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move completed: OK
I20240502 14:16:37.020833 31448 rebalancer_tool.cc:1136] tablet f4dbd9bc62484dd1998080d289db098f: '9329e8a7a2d1453d90b76b5965315837' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move completed: Incomplete: tablet f4dbd9bc62484dd1998080d289db098f, TS 9329e8a7a2d1453d90b76b5965315837 -> TS d653f68cb53b4ed79d5e688987ed8d4c move failed, target replica disappeared
I20240502 14:16:37.176192 31448 rebalancer_tool.cc:1136] tablet 71f047f74165489bacb4f72ab981b447: '1b3c20f7bfd548769f6bd40148590b74' -> 'b7dacf388234446b9a11ce750978ca75' move completed: OK
I20240502 14:16:37.245098 31448 rebalancer_tool.cc:1136] tablet 39835ac7b2bb4c0a8c4b43b010b09706: '139e612d62744c02bcb5653b7aa1dc29' -> 'ede5e1f11f254824a335e1e43f40b6d1' move completed: OK
I20240502 14:16:37.540374 31448 rebalancer_tool.cc:1136] tablet e6a0a881fcec43b291148f071e4474fc: 'dbabfd51061b4e509d907a3622febd2e' -> '3bb852e0a21446bbbe824bdefb16036c' move completed: Incomplete: tablet e6a0a881fcec43b291148f071e4474fc, TS dbabfd51061b4e509d907a3622febd2e -> TS 3bb852e0a21446bbbe824bdefb16036c move failed, target replica disappeared
I20240502 14:16:37.607360 31448 rebalancer_tool.cc:1136] tablet d3221d1a1b244a27abc9ee26e0a6edb1: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move completed: OK
I20240502 14:16:37.636519 31448 rebalancer_tool.cc:1136] tablet fd4e4241d42d483ab6203606a4d92141: '1b3c20f7bfd548769f6bd40148590b74' -> '' move completed: OK
I20240502 14:16:37.708407 31448 rebalancer_tool.cc:1084] tablet 302978e1f6034b3693b3a0abacdc494e: 'b7dacf388234446b9a11ce750978ca75' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move ignored: Not found: Peer with uuid b7dacf388234446b9a11ce750978ca75 not found in consensus config
I20240502 14:16:37.708794 31448 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): unacceptable health status UNAVAILABLE
I20240502 14:16:38.509862 31442 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3cgK4A/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:16:18.193742 31444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:18.194203 31444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:18.206159 31444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:16:20.590723 31444 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:16:22.278838 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:22.279392 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:22.279596 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:22.279753 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a61f61a448354326a1834e5e87d7d2b5: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:22.279861 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c623a6a1d9954161b2d78dcce7d02b20: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:22.280038 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:22.280156 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:22.280337 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:22.280527 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:22.280649 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:22.280992 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c8d2fd0bad3e4c45862835f106e6b44b: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server b7dacf388234446b9a11ce750978ca75
W20240502 14:16:22.281140 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:22.281280 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c72cf765d124fa69d0f3f17b1247b14: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:22.281397 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5d3d62d9036a4b349caec86fc6dcf0da: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:22.457852 31444 rebalancer_tool.cc:1074] tablet 3fde0741532a4067a4806f4bc0b0d465: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:22.761870 31444 rebalancer_tool.cc:1074] tablet 3901ce6537fd48ba861fba13767d8aa4: '1b3c20f7bfd548769f6bd40148590b74' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:23.013525 31444 rebalancer_tool.cc:1074] tablet 1f17889f58a34bc0afa40a246e5debc6: '9329e8a7a2d1453d90b76b5965315837' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:23.413897 31444 rebalancer_tool.cc:1074] tablet 981ec6067d624100bddba4383b417c50: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:23.869839 31444 rebalancer_tool.cc:1074] tablet 093e29d30b2646d68e84b8b9ca86ca9e: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:23.965842 31444 rebalancer_tool.cc:1074] tablet a0fcde0e7cbe41f1a521e669b798aa99: '1b3c20f7bfd548769f6bd40148590b74' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:24.095108 31444 rebalancer_tool.cc:1074] tablet 63690fff28c84670b15ed02ba1356e33: '9329e8a7a2d1453d90b76b5965315837' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:24.381875 31444 rebalancer_tool.cc:1074] tablet e89c7a017200449abcd5d9b9c2bcc912: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:25.030205 31444 rebalancer_tool.cc:1074] tablet 44d94461e22c4ae182afa2a15f3db198: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:25.304277 31444 rebalancer_tool.cc:1074] tablet a230503874a348f4a19647179fe25123: '1b3c20f7bfd548769f6bd40148590b74' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:25.979439 31444 rebalancer_tool.cc:1074] tablet 302978e1f6034b3693b3a0abacdc494e: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:26.713364 31444 rebalancer_tool.cc:1074] tablet 078ffca14316466aa3906d2ff0567c70: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:27.107105 31444 rebalancer_tool.cc:1074] tablet 96fb0fc5919d42d7bedd770945935b42: 'dbabfd51061b4e509d907a3622febd2e' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:27.699347 31444 rebalancer_tool.cc:1074] tablet b7bfc3bb327b4407ae1aa7742f4b18de: '1b3c20f7bfd548769f6bd40148590b74' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move scheduled
I20240502 14:16:28.044935 31444 rebalancer_tool.cc:1084] tablet fd4e4241d42d483ab6203606a4d92141: '9329e8a7a2d1453d90b76b5965315837' -> 'b7dacf388234446b9a11ce750978ca75' move ignored: Illegal state: tablet fd4e4241d42d483ab6203606a4d92141: replica b7dacf388234446b9a11ce750978ca75 is already present
I20240502 14:16:28.045109 31444 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:16:31.333554 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server b7dacf388234446b9a11ce750978ca75 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:31.333786 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c36807ea6a59441696e14f8d2dbf3053: could not find any suitable replica to move from server 9329e8a7a2d1453d90b76b5965315837 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:31.334045 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:31.345651 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b097cc0423da401bbcf8fbbdc9ccd9aa: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
W20240502 14:16:31.345880 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 97eb0e50641648b8b13df0a289a86046: could not find any suitable replica to move from server dbabfd51061b4e509d907a3622febd2e to server d653f68cb53b4ed79d5e688987ed8d4c
W20240502 14:16:31.346009 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd071eb2b4d148b7b60d6e9293fbcdd8: could not find any suitable replica to move from server 1b3c20f7bfd548769f6bd40148590b74 to server 3bb852e0a21446bbbe824bdefb16036c
W20240502 14:16:31.346113 31444 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 88709dc4b0224a92a8aa5ffdaf78ebf8: could not find any suitable replica to move from server 139e612d62744c02bcb5653b7aa1dc29 to server ede5e1f11f254824a335e1e43f40b6d1
I20240502 14:16:31.556186 31444 rebalancer_tool.cc:1074] tablet d95b84899c994d45a569d5a4dda124ed: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:31.853574 31444 rebalancer_tool.cc:1074] tablet 39835ac7b2bb4c0a8c4b43b010b09706: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move scheduled
I20240502 14:16:32.156909 31444 rebalancer_tool.cc:1074] tablet e6a0a881fcec43b291148f071e4474fc: '1b3c20f7bfd548769f6bd40148590b74' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:32.602334 31444 rebalancer_tool.cc:1074] tablet e66f9bb135bf4007b1864b63a389fac3: 'd653f68cb53b4ed79d5e688987ed8d4c' -> '139e612d62744c02bcb5653b7aa1dc29' move scheduled
I20240502 14:16:33.132113 31444 rebalancer_tool.cc:1074] tablet a6eb9dee735a4115874ef93636a559ab: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move scheduled
I20240502 14:16:33.717005 31444 rebalancer_tool.cc:1136] tablet 093e29d30b2646d68e84b8b9ca86ca9e: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move completed: OK
I20240502 14:16:33.810264 31444 rebalancer_tool.cc:1136] tablet 3fde0741532a4067a4806f4bc0b0d465: 'dbabfd51061b4e509d907a3622febd2e' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move completed: OK
I20240502 14:16:33.995513 31444 rebalancer_tool.cc:1136] tablet a0fcde0e7cbe41f1a521e669b798aa99: '1b3c20f7bfd548769f6bd40148590b74' -> '3bb852e0a21446bbbe824bdefb16036c' move completed: OK
I20240502 14:16:34.132369 31444 rebalancer_tool.cc:1136] tablet 1f17889f58a34bc0afa40a246e5debc6: '9329e8a7a2d1453d90b76b5965315837' -> 'ede5e1f11f254824a335e1e43f40b6d1' move completed: OK
I20240502 14:16:34.391162 31444 rebalancer_tool.cc:1136] tablet 078ffca14316466aa3906d2ff0567c70: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move completed: OK
I20240502 14:16:34.580524 31444 rebalancer_tool.cc:1136] tablet 3901ce6537fd48ba861fba13767d8aa4: '1b3c20f7bfd548769f6bd40148590b74' -> '3bb852e0a21446bbbe824bdefb16036c' move completed: OK
I20240502 14:16:34.773082 31444 rebalancer_tool.cc:1136] tablet 63690fff28c84670b15ed02ba1356e33: '9329e8a7a2d1453d90b76b5965315837' -> 'ede5e1f11f254824a335e1e43f40b6d1' move completed: OK
I20240502 14:16:34.970589 31444 rebalancer_tool.cc:1136] tablet e89c7a017200449abcd5d9b9c2bcc912: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move completed: OK
I20240502 14:16:35.374152 31444 rebalancer_tool.cc:1136] tablet a230503874a348f4a19647179fe25123: '1b3c20f7bfd548769f6bd40148590b74' -> 'ede5e1f11f254824a335e1e43f40b6d1' move completed: OK
I20240502 14:16:36.307121 31444 rebalancer_tool.cc:1136] tablet e6a0a881fcec43b291148f071e4474fc: '1b3c20f7bfd548769f6bd40148590b74' -> 'ede5e1f11f254824a335e1e43f40b6d1' move completed: OK
I20240502 14:16:36.980185 31444 rebalancer_tool.cc:1136] tablet a6eb9dee735a4115874ef93636a559ab: 'dbabfd51061b4e509d907a3622febd2e' -> 'b7dacf388234446b9a11ce750978ca75' move completed: OK
I20240502 14:16:37.054656 31444 rebalancer_tool.cc:1136] tablet 302978e1f6034b3693b3a0abacdc494e: '9329e8a7a2d1453d90b76b5965315837' -> '3bb852e0a21446bbbe824bdefb16036c' move completed: OK
I20240502 14:16:37.402135 31444 rebalancer_tool.cc:1136] tablet 981ec6067d624100bddba4383b417c50: '139e612d62744c02bcb5653b7aa1dc29' -> 'b7dacf388234446b9a11ce750978ca75' move completed: OK
I20240502 14:16:37.557160 31444 rebalancer_tool.cc:1136] tablet b7bfc3bb327b4407ae1aa7742f4b18de: '1b3c20f7bfd548769f6bd40148590b74' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move completed: OK
I20240502 14:16:37.637928 31444 rebalancer_tool.cc:1074] tablet f4dbd9bc62484dd1998080d289db098f: '1b3c20f7bfd548769f6bd40148590b74' -> 'ede5e1f11f254824a335e1e43f40b6d1' move scheduled
I20240502 14:16:37.719133 31444 rebalancer_tool.cc:1084] tablet d67ee220c86f4fc7bc1db8ae08d668d1: '9329e8a7a2d1453d90b76b5965315837' -> 'd653f68cb53b4ed79d5e688987ed8d4c' move ignored: Illegal state: tablet d67ee220c86f4fc7bc1db8ae08d668d1: replica d653f68cb53b4ed79d5e688987ed8d4c is already present
I20240502 14:16:37.719259 31444 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): unacceptable health status UNAVAILABLE
E20240502 14:16:38.510612 21737 external_mini_cluster.cc:943] Process with UUID b7dacf388234446b9a11ce750978ca75 has crashed
E20240502 14:16:38.510744 21737 external_mini_cluster.cc:943] Process with UUID d653f68cb53b4ed79d5e688987ed8d4c has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 14:16:38.512318 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 28975
I20240502 14:16:38.519941 29095 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:16:38.548424 29255 consensus_peers.cc:485] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:38.569689 29121 connection.cc:535] client connection to 127.21.58.69:36535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:38.569869 29385 connection.cc:535] client connection to 127.21.58.69:36535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:38.570649 29254 connection.cc:535] client connection to 127.21.58.69:36535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:38.571296 29385 consensus_peers.cc:462] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111)
W20240502 14:16:38.571334 29121 consensus_peers.cc:485] T 981ec6067d624100bddba4383b417c50 P dbabfd51061b4e509d907a3622febd2e -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.571403 29385 consensus_peers.cc:485] T 366dbe98fccd42cb8e369876230984fe P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.571491 29385 consensus_peers.cc:485] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.571933 29254 consensus_peers.cc:485] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.572042 29254 consensus_peers.cc:485] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.572153 29254 consensus_peers.cc:485] T 1f17889f58a34bc0afa40a246e5debc6 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.593381 29385 consensus_peers.cc:485] T 302978e1f6034b3693b3a0abacdc494e P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.598278 29123 consensus_peers.cc:485] T e89c7a017200449abcd5d9b9c2bcc912 P dbabfd51061b4e509d907a3622febd2e -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:38.614409 29385 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.709353 28902 catalog_manager.cc:4573] TS d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96fb0fc5919d42d7bedd770945935b42: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
W20240502 14:16:38.769578 29385 consensus_peers.cc:462] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111)
W20240502 14:16:38.769996 29386 consensus_peers.cc:485] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:38.791623 29254 consensus_peers.cc:485] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Couldn't send request to peer ede5e1f11f254824a335e1e43f40b6d1. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.804929 28990 connection.cc:698] client connection to 127.21.58.69:36535 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20240502 14:16:38.807475 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 28975
W20240502 14:16:38.857793 29256 connection.cc:535] client connection to 127.21.58.65:46783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:38.857793 29255 connection.cc:535] server connection from 127.21.58.65:44775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:38.857941 29125 connection.cc:535] server connection from 127.21.58.65:41513 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:38.858340 29124 connection.cc:535] client connection to 127.21.58.65:46783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:38.858343 29387 connection.cc:535] client connection to 127.21.58.65:46783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:38.858834 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 29106
W20240502 14:16:38.859548 29256 consensus_peers.cc:485] T c4818cd7d9b44111a04a0d32a4580cdf P 1b3c20f7bfd548769f6bd40148590b74 -> Peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): Couldn't send request to peer 9329e8a7a2d1453d90b76b5965315837. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:46783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.859673 29256 consensus_peers.cc:485] T d67ee220c86f4fc7bc1db8ae08d668d1 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): Couldn't send request to peer 9329e8a7a2d1453d90b76b5965315837. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:46783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.860213 29387 consensus_peers.cc:485] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): Couldn't send request to peer 9329e8a7a2d1453d90b76b5965315837. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:46783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.860411 29124 consensus_peers.cc:485] T e66f9bb135bf4007b1864b63a389fac3 P dbabfd51061b4e509d907a3622febd2e -> Peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): Couldn't send request to peer 9329e8a7a2d1453d90b76b5965315837. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:46783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:38.861254 29386 consensus_peers.cc:485] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:38.865087 29229 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:16:38.877811 29257 consensus_peers.cc:462] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
W20240502 14:16:38.926386 29256 consensus_peers.cc:485] T a00d9c46bfc64a4085715e4f764b8c96 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783): Couldn't send request to peer 9329e8a7a2d1453d90b76b5965315837. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:46783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:39.045523 29255 consensus_peers.cc:485] T 432dc9a199294b748414520a2c49a674 P 1b3c20f7bfd548769f6bd40148590b74 -> Peer b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703): Couldn't send request to peer b7dacf388234446b9a11ce750978ca75. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.71:44703: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:39.112035 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 29106
W20240502 14:16:39.118000 31062 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:39.151997 31062 connection.cc:535] server connection from 127.21.58.66:43931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:39.152254 31062 connection.cc:535] client connection to 127.21.58.66:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:39.152807 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 29239
W20240502 14:16:39.152809 28901 connection.cc:535] server connection from 127.21.58.66:49133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:39.153565 29388 consensus_peers.cc:485] T 63690fff28c84670b15ed02ba1356e33 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): Couldn't send request to peer dbabfd51061b4e509d907a3622febd2e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:39.153597 31062 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c -> Peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): Couldn't send request to peer dbabfd51061b4e509d907a3622febd2e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:39.153702 29388 consensus_peers.cc:485] T e6a0a881fcec43b291148f071e4474fc P 139e612d62744c02bcb5653b7aa1dc29 -> Peer dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753): Couldn't send request to peer dbabfd51061b4e509d907a3622febd2e. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.66:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:39.157428 29388 consensus_peers.cc:485] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:39.159265 29359 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:16:39.259342 29388 consensus_peers.cc:485] T a6eb9dee735a4115874ef93636a559ab P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:39.286664 29388 consensus_peers.cc:485] T 71f047f74165489bacb4f72ab981b447 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): Couldn't send request to peer d653f68cb53b4ed79d5e688987ed8d4c. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:39.352669 29385 consensus_peers.cc:462] T f4dbd9bc62484dd1998080d289db098f P 139e612d62744c02bcb5653b7aa1dc29 -> Peer ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.21.58.69:36535: connect: Connection refused (error 111)
I20240502 14:16:39.396025 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 29239
W20240502 14:16:39.435386 29385 connection.cc:535] client connection to 127.21.58.67:40039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:39.435825 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 29370
W20240502 14:16:39.436394 31059 consensus_peers.cc:485] T 44d94461e22c4ae182afa2a15f3db198 P 3bb852e0a21446bbbe824bdefb16036c -> Peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): Couldn't send request to peer 1b3c20f7bfd548769f6bd40148590b74. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.67:40039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:39.436540 29385 consensus_peers.cc:485] T d3221d1a1b244a27abc9ee26e0a6edb1 P 139e612d62744c02bcb5653b7aa1dc29 -> Peer 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039): Couldn't send request to peer 1b3c20f7bfd548769f6bd40148590b74. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.67:40039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:39.441382 29491 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:39.692718 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 29370
W20240502 14:16:39.735306 31060 connection.cc:535] server connection from 127.21.58.68:52761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:39.735813 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 31042
I20240502 14:16:39.747264 31167 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:16:39.771955 28902 catalog_manager.cc:4573] TS d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96fb0fc5919d42d7bedd770945935b42: Network error: Client connection negotiation failed: client connection to 127.21.58.72:36549: connect: Connection refused (error 111)
I20240502 14:16:39.931485 2078 raft_consensus.cc:482] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9329e8a7a2d1453d90b76b5965315837)
I20240502 14:16:39.931731 2078 raft_consensus.cc:504] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "ede5e1f11f254824a335e1e43f40b6d1" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 36535 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:39.932741 2078 leader_election.cc:288] T 39835ac7b2bb4c0a8c4b43b010b09706 P 3bb852e0a21446bbbe824bdefb16036c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), ede5e1f11f254824a335e1e43f40b6d1 (127.21.58.69:36535)
I20240502 14:16:39.932974 2078 raft_consensus.cc:482] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9329e8a7a2d1453d90b76b5965315837)
I20240502 14:16:39.933144 2078 raft_consensus.cc:504] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b3c20f7bfd548769f6bd40148590b74" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40039 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "d653f68cb53b4ed79d5e688987ed8d4c" member_type: VOTER last_known_addr { host: "127.21.58.72" port: 36549 } attrs { promote: false } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } }
I20240502 14:16:39.933801 2078 leader_election.cc:288] T 093e29d30b2646d68e84b8b9ca86ca9e P 3bb852e0a21446bbbe824bdefb16036c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b3c20f7bfd548769f6bd40148590b74 (127.21.58.67:40039), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), d653f68cb53b4ed79d5e688987ed8d4c (127.21.58.72:36549)
I20240502 14:16:39.933960 2078 raft_consensus.cc:482] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dbabfd51061b4e509d907a3622febd2e)
I20240502 14:16:39.934124 2078 raft_consensus.cc:504] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "139e612d62744c02bcb5653b7aa1dc29" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:39.935210 2079 raft_consensus.cc:482] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9329e8a7a2d1453d90b76b5965315837)
I20240502 14:16:39.935317 2078 leader_election.cc:288] T d95b84899c994d45a569d5a4dda124ed P 3bb852e0a21446bbbe824bdefb16036c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 139e612d62744c02bcb5653b7aa1dc29 (127.21.58.68:46509), b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703)
I20240502 14:16:39.935411 2079 raft_consensus.cc:504] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dbabfd51061b4e509d907a3622febd2e" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 44753 } } peers { permanent_uuid: "9329e8a7a2d1453d90b76b5965315837" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 46783 } attrs { replace: true } } peers { permanent_uuid: "3bb852e0a21446bbbe824bdefb16036c" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 41819 } attrs { promote: false } } peers { permanent_uuid: "b7dacf388234446b9a11ce750978ca75" member_type: VOTER last_known_addr { host: "127.21.58.71" port: 44703 } attrs { promote: false } }
I20240502 14:16:39.936616 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 31042
I20240502 14:16:39.936589 2079 leader_election.cc:288] T 3901ce6537fd48ba861fba13767d8aa4 P 3bb852e0a21446bbbe824bdefb16036c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dbabfd51061b4e509d907a3622febd2e (127.21.58.66:44753), 9329e8a7a2d1453d90b76b5965315837 (127.21.58.65:46783), b7dacf388234446b9a11ce750978ca75 (127.21.58.71:44703)
I20240502 14:16:39.968181 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 28884
I20240502 14:16:39.969188 28943 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:40.225800 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 28884
2024-05-02T14:16:40Z chronyd exiting
I20240502 14:16:40.286314 21737 test_util.cc:175] -----------------------------------------------
I20240502 14:16:40.286399 21737 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714659268621181-21737-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (47998 ms)
[----------] 1 test from ConcurrentRebalancersTest (47998 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240502 14:16:40.289042 21737 test_util.cc:268] Using random seed: 1081543417
I20240502 14:16:40.290820 21737 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:16:40.290880 21737 ts_itest-base.cc:113] --------------
I20240502 14:16:40.290928 21737 ts_itest-base.cc:114] 4 tablet servers
I20240502 14:16:40.290984 21737 ts_itest-base.cc:115] 1 replicas per TS
I20240502 14:16:40.291023 21737 ts_itest-base.cc:116] --------------
2024-05-02T14:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:16:40Z Disabled control of system clock
I20240502 14:16:40.340520 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:42477
--webserver_interface=127.21.58.126
--webserver_port=0
--builtin_ntp_servers=127.21.58.84:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.58.126:42477
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 14:16:40.669878 2098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:40.670166 2098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:40.670253 2098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:40.677436 2098 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:40.677546 2098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:16:40.677611 2098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:40.677666 2098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:16:40.677709 2098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:16:40.686609 2098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.58.126:42477
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:42477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.58.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:40.687955 2098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:40.689649 2098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:40.700378 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:40.700562 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:40.701122 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:40.701792 2098 server_base.cc:1034] running on GCE node
I20240502 14:16:40.702479 2098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:40.703810 2098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:40.705011 2098 hybrid_clock.cc:648] HybridClock initialized: now 1714659400704968 us; error 33 us; skew 500 ppm
I20240502 14:16:40.705353 2098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:40.707525 2098 webserver.cc:460] Webserver started at http://127.21.58.126:36391/ using document root <none> and password file <none>
I20240502 14:16:40.708210 2098 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:40.708314 2098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:40.708613 2098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:40.711113 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8e34e188582247178f4c1687d1bdeb0c"
format_stamp: "Formatted at 2024-05-02 14:16:40 on dist-test-slave-wtgm"
I20240502 14:16:40.711889 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8e34e188582247178f4c1687d1bdeb0c"
format_stamp: "Formatted at 2024-05-02 14:16:40 on dist-test-slave-wtgm"
I20240502 14:16:40.716742 2098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:16:40.719774 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:40.720906 2098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:40.721063 2098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8e34e188582247178f4c1687d1bdeb0c"
format_stamp: "Formatted at 2024-05-02 14:16:40 on dist-test-slave-wtgm"
I20240502 14:16:40.721247 2098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:40.735034 2098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:40.735821 2098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:40.736104 2098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:40.753870 2098 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.126:42477
I20240502 14:16:40.753888 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.126:42477 every 8 connection(s)
I20240502 14:16:40.755283 2098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:16:40.756626 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 2098
I20240502 14:16:40.756865 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:16:40.760084 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:40.770922 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c: Bootstrap starting.
I20240502 14:16:40.773787 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:40.774708 2164 log.cc:824] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:40.777206 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c: No bootstrap required, opened a new log
I20240502 14:16:40.783361 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34e188582247178f4c1687d1bdeb0c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42477 } }
I20240502 14:16:40.783676 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:40.783753 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e34e188582247178f4c1687d1bdeb0c, State: Initialized, Role: FOLLOWER
I20240502 14:16:40.784267 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34e188582247178f4c1687d1bdeb0c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42477 } }
I20240502 14:16:40.784443 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:40.784538 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:40.784662 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:40.786800 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34e188582247178f4c1687d1bdeb0c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42477 } }
I20240502 14:16:40.787274 2164 leader_election.cc:302] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e34e188582247178f4c1687d1bdeb0c; no voters:
I20240502 14:16:40.787789 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:40.787917 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:40.788658 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [term 1 LEADER]: Becoming Leader. State: Replica: 8e34e188582247178f4c1687d1bdeb0c, State: Running, Role: LEADER
I20240502 14:16:40.789160 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34e188582247178f4c1687d1bdeb0c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42477 } }
I20240502 14:16:40.789839 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:16:40.791677 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e34e188582247178f4c1687d1bdeb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34e188582247178f4c1687d1bdeb0c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42477 } } }
I20240502 14:16:40.791949 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [sys.catalog]: This master's current role is: LEADER
I20240502 14:16:40.792465 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e34e188582247178f4c1687d1bdeb0c. Latest consensus state: current_term: 1 leader_uuid: "8e34e188582247178f4c1687d1bdeb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34e188582247178f4c1687d1bdeb0c" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 42477 } } }
I20240502 14:16:40.792740 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c [sys.catalog]: This master's current role is: LEADER
I20240502 14:16:40.793524 2176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:16:40.797672 2176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:16:40.813958 2176 catalog_manager.cc:1347] Generated new cluster ID: 94929df1eb0e4eeeaf45001c9f814b59
I20240502 14:16:40.814126 2176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:16:40.825342 2176 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:16:40.826978 2176 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:16:40.848940 2176 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8e34e188582247178f4c1687d1bdeb0c: Generated new TSK 0
I20240502 14:16:40.849973 2176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:16:40.861433 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.65:0
--local_ip_for_outbound_sockets=127.21.58.65
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--builtin_ntp_servers=127.21.58.84:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:16:41.190995 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:41.191280 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:41.191407 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:41.198283 2190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:41.198388 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:41.198544 2190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.65
I20240502 14:16:41.206666 2190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.65
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:41.207996 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:41.209658 2190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:41.220964 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:41.221408 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:41.222050 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:41.222236 2190 server_base.cc:1034] running on GCE node
I20240502 14:16:41.222751 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:41.223610 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:41.224834 2190 hybrid_clock.cc:648] HybridClock initialized: now 1714659401224791 us; error 36 us; skew 500 ppm
I20240502 14:16:41.225133 2190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:41.227421 2190 webserver.cc:460] Webserver started at http://127.21.58.65:42735/ using document root <none> and password file <none>
I20240502 14:16:41.228111 2190 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:41.228227 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:41.228538 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:41.231217 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2c3e885d68564b61ba3401a80ee146e2"
format_stamp: "Formatted at 2024-05-02 14:16:41 on dist-test-slave-wtgm"
I20240502 14:16:41.231974 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2c3e885d68564b61ba3401a80ee146e2"
format_stamp: "Formatted at 2024-05-02 14:16:41 on dist-test-slave-wtgm"
I20240502 14:16:41.236879 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 14:16:41.240249 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:41.241456 2190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:16:41.241602 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2c3e885d68564b61ba3401a80ee146e2"
format_stamp: "Formatted at 2024-05-02 14:16:41 on dist-test-slave-wtgm"
I20240502 14:16:41.241788 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:41.281762 2190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:41.282583 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:41.282876 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:41.283836 2190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:41.285219 2190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:41.285311 2190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:41.285421 2190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:41.285491 2190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:41.316912 2190 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.65:42409
I20240502 14:16:41.316982 2318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.65:42409 every 8 connection(s)
I20240502 14:16:41.318182 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:16:41.319413 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 2190
I20240502 14:16:41.319634 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:16:41.323365 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.66:0
--local_ip_for_outbound_sockets=127.21.58.66
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--builtin_ntp_servers=127.21.58.84:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:41.330991 2319 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42477
I20240502 14:16:41.331315 2319 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:41.332123 2319 heartbeater.cc:511] Master 127.21.58.126:42477 requested a full tablet report, sending...
I20240502 14:16:41.334245 2129 ts_manager.cc:194] Registered new tserver with Master: 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:41.336441 2129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.65:56511
W20240502 14:16:41.653149 2323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:41.653425 2323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:41.653512 2323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:41.661079 2323 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:41.661195 2323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:41.661314 2323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.66
I20240502 14:16:41.670154 2323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.66
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:41.671506 2323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:41.673461 2323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:41.685037 2328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:41.685653 2329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:41.686178 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:41.686834 2323 server_base.cc:1034] running on GCE node
I20240502 14:16:41.687441 2323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:41.688328 2323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:41.689514 2323 hybrid_clock.cc:648] HybridClock initialized: now 1714659401689461 us; error 43 us; skew 500 ppm
I20240502 14:16:41.689852 2323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:41.692032 2323 webserver.cc:460] Webserver started at http://127.21.58.66:39271/ using document root <none> and password file <none>
I20240502 14:16:41.692756 2323 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:41.692874 2323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:41.693157 2323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:41.695956 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "35dad2fd4bed4e71a86573ec6096908a"
format_stamp: "Formatted at 2024-05-02 14:16:41 on dist-test-slave-wtgm"
I20240502 14:16:41.696671 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "35dad2fd4bed4e71a86573ec6096908a"
format_stamp: "Formatted at 2024-05-02 14:16:41 on dist-test-slave-wtgm"
I20240502 14:16:41.701503 2323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:16:41.704744 2337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:41.705927 2323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20240502 14:16:41.706101 2323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "35dad2fd4bed4e71a86573ec6096908a"
format_stamp: "Formatted at 2024-05-02 14:16:41 on dist-test-slave-wtgm"
I20240502 14:16:41.706279 2323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:41.738267 2323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:41.739192 2323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:41.739471 2323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:41.740401 2323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:41.741878 2323 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:41.741977 2323 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:41.742092 2323 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:41.742163 2323 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:41.776360 2323 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.66:36889
I20240502 14:16:41.776424 2450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.66:36889 every 8 connection(s)
I20240502 14:16:41.777630 2323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:16:41.785127 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 2323
I20240502 14:16:41.785297 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:16:41.790491 2451 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42477
I20240502 14:16:41.790854 2451 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:41.791954 2451 heartbeater.cc:511] Master 127.21.58.126:42477 requested a full tablet report, sending...
I20240502 14:16:41.793525 2129 ts_manager.cc:194] Registered new tserver with Master: 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:41.794277 2129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.66:52691
I20240502 14:16:41.834080 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:0
--local_ip_for_outbound_sockets=127.21.58.67
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--builtin_ntp_servers=127.21.58.84:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:16:42.173888 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:42.174172 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:42.174307 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:42.181329 2454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:42.181457 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:42.181607 2454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:16:42.190044 2454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:42.191443 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:42.193070 2454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:42.203889 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:42.204497 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:42.205119 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:42.205564 2454 server_base.cc:1034] running on GCE node
I20240502 14:16:42.206147 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:42.206995 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:42.208191 2454 hybrid_clock.cc:648] HybridClock initialized: now 1714659402208151 us; error 29 us; skew 500 ppm
I20240502 14:16:42.208469 2454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:42.210665 2454 webserver.cc:460] Webserver started at http://127.21.58.67:43421/ using document root <none> and password file <none>
I20240502 14:16:42.211380 2454 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:42.211477 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:42.211737 2454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:42.214121 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "948dd66e071d426c95807b11c23599ce"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:42.214812 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "948dd66e071d426c95807b11c23599ce"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:42.219547 2454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:16:42.222616 2469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:42.223862 2454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:42.223989 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "948dd66e071d426c95807b11c23599ce"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:42.224148 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:42.243901 2454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:42.244733 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:42.244969 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:42.245935 2454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:42.247440 2454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:42.247525 2454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:42.247604 2454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:42.247663 2454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:42.280381 2454 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:38343
I20240502 14:16:42.280426 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:38343 every 8 connection(s)
I20240502 14:16:42.281762 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:16:42.283394 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 2454
I20240502 14:16:42.283581 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:16:42.287566 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:0
--local_ip_for_outbound_sockets=127.21.58.68
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--builtin_ntp_servers=127.21.58.84:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:42.296124 2583 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42477
I20240502 14:16:42.296442 2583 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:42.297410 2583 heartbeater.cc:511] Master 127.21.58.126:42477 requested a full tablet report, sending...
I20240502 14:16:42.298954 2129 ts_manager.cc:194] Registered new tserver with Master: 948dd66e071d426c95807b11c23599ce (127.21.58.67:38343)
I20240502 14:16:42.299947 2129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:37813
I20240502 14:16:42.339247 2319 heartbeater.cc:503] Master 127.21.58.126:42477 was elected leader, sending a full tablet report...
W20240502 14:16:42.630146 2586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:42.630429 2586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:42.630564 2586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:42.637799 2586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:42.637931 2586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:42.638088 2586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:16:42.646839 2586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:42477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:42.648142 2586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:42.649907 2586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:42.660842 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:42.661458 2594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:42.662101 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:42.662746 2586 server_base.cc:1034] running on GCE node
I20240502 14:16:42.663297 2586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:42.664088 2586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:42.665292 2586 hybrid_clock.cc:648] HybridClock initialized: now 1714659402665245 us; error 42 us; skew 500 ppm
I20240502 14:16:42.665580 2586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:42.667681 2586 webserver.cc:460] Webserver started at http://127.21.58.68:34585/ using document root <none> and password file <none>
I20240502 14:16:42.668349 2586 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:42.668469 2586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:42.668758 2586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:42.671175 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "96bce754191b4a478796de231f93fb34"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:42.671903 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "96bce754191b4a478796de231f93fb34"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:42.676760 2586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 14:16:42.679906 2600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:42.681186 2586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:16:42.681341 2586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "96bce754191b4a478796de231f93fb34"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:42.681531 2586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:42.708978 2586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:42.709952 2586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:42.710263 2586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:42.711612 2586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:42.713244 2586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:42.713348 2586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:42.713482 2586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:42.713562 2586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:42.745215 2586 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:34509
I20240502 14:16:42.745355 2712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:34509 every 8 connection(s)
I20240502 14:16:42.746723 2586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:16:42.752305 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 2586
I20240502 14:16:42.752487 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:16:42.760556 2713 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42477
I20240502 14:16:42.760851 2713 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:42.761610 2713 heartbeater.cc:511] Master 127.21.58.126:42477 requested a full tablet report, sending...
I20240502 14:16:42.762889 2129 ts_manager.cc:194] Registered new tserver with Master: 96bce754191b4a478796de231f93fb34 (127.21.58.68:34509)
I20240502 14:16:42.763672 2129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:53905
I20240502 14:16:42.768100 21737 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 14:16:42.783033 2129 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45712:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:16:42.797205 2451 heartbeater.cc:503] Master 127.21.58.126:42477 was elected leader, sending a full tablet report...
I20240502 14:16:42.810529 2385 tablet_service.cc:1467] Processing CreateTablet for tablet 821c64a4d89e4dd88d21397253348a10 (DEFAULT_TABLE table=TestTable [id=22f012de21434cbebe8520ac600a51a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:42.811966 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821c64a4d89e4dd88d21397253348a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:42.821482 2727 tablet_bootstrap.cc:492] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a: Bootstrap starting.
I20240502 14:16:42.824644 2727 tablet_bootstrap.cc:654] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:42.825858 2727 log.cc:824] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:42.828423 2727 tablet_bootstrap.cc:492] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a: No bootstrap required, opened a new log
I20240502 14:16:42.828693 2727 ts_tablet_manager.cc:1367] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 14:16:42.835019 2727 raft_consensus.cc:348] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:42.835386 2727 raft_consensus.cc:374] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:42.835480 2727 raft_consensus.cc:729] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Initialized, Role: FOLLOWER
I20240502 14:16:42.836097 2727 consensus_queue.cc:260] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:42.836298 2727 raft_consensus.cc:388] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:42.836418 2727 raft_consensus.cc:482] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:42.836565 2727 raft_consensus.cc:3037] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:42.838599 2727 raft_consensus.cc:504] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:42.839048 2727 leader_election.cc:302] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:42.839612 2727 leader_election.cc:288] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:42.839771 2729 raft_consensus.cc:2781] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:42.840387 2729 raft_consensus.cc:686] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Leader. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:42.840894 2729 consensus_queue.cc:237] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:42.841373 2727 ts_tablet_manager.cc:1398] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20240502 14:16:42.846256 2128 catalog_manager.cc:5484] T 821c64a4d89e4dd88d21397253348a10 P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66). New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:42.866780 21737 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 14:16:42.868726 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2c3e885d68564b61ba3401a80ee146e2 to finish bootstrapping
I20240502 14:16:42.882725 21737 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 35dad2fd4bed4e71a86573ec6096908a to finish bootstrapping
I20240502 14:16:42.889575 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 948dd66e071d426c95807b11c23599ce to finish bootstrapping
I20240502 14:16:42.897190 21737 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 96bce754191b4a478796de231f93fb34 to finish bootstrapping
I20240502 14:16:42.907999 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 2454
I20240502 14:16:42.908946 2578 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:42.987432 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 2454
I20240502 14:16:43.004920 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 2586
I20240502 14:16:43.005973 2708 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:43.083134 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 2586
I20240502 14:16:46.853226 2128 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45712:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:46.871680 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9f0f44361a48799b2c978b1585bd1d (DEFAULT_TABLE table=rebalance_test_table_0 [id=77bc6f9f1fc541c69d3501c0664659d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:46.871591 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 36ccbeda4b0f4efbbb8edc2b571e94dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=77bc6f9f1fc541c69d3501c0664659d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:46.871618 2251 tablet_service.cc:1467] Processing CreateTablet for tablet 357f9a3dd69240c0aaff828e372210f0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=77bc6f9f1fc541c69d3501c0664659d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:46.873210 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9f0f44361a48799b2c978b1585bd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:46.873634 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357f9a3dd69240c0aaff828e372210f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:46.873912 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ccbeda4b0f4efbbb8edc2b571e94dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:46.884414 2748 tablet_bootstrap.cc:492] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap starting.
I20240502 14:16:46.887169 2748 tablet_bootstrap.cc:654] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:46.888142 2748 log.cc:824] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:46.890540 2748 tablet_bootstrap.cc:492] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: No bootstrap required, opened a new log
I20240502 14:16:46.890756 2748 ts_tablet_manager.cc:1367] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:16:46.896819 2748 raft_consensus.cc:348] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.897130 2748 raft_consensus.cc:374] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:46.897229 2748 raft_consensus.cc:729] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Initialized, Role: FOLLOWER
I20240502 14:16:46.897846 2748 consensus_queue.cc:260] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.898046 2748 raft_consensus.cc:388] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:46.898169 2748 raft_consensus.cc:482] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:46.898309 2748 raft_consensus.cc:3037] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:46.900256 2748 raft_consensus.cc:504] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.900687 2748 leader_election.cc:302] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2; no voters:
I20240502 14:16:46.901225 2748 leader_election.cc:288] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:46.901364 2750 raft_consensus.cc:2781] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:46.902920 2748 ts_tablet_manager.cc:1398] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20240502 14:16:46.903304 2748 tablet_bootstrap.cc:492] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap starting.
I20240502 14:16:46.904805 2748 tablet_bootstrap.cc:654] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:46.906776 2748 tablet_bootstrap.cc:492] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: No bootstrap required, opened a new log
I20240502 14:16:46.906754 2750 raft_consensus.cc:686] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:46.906949 2748 ts_tablet_manager.cc:1367] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:16:46.907315 2750 consensus_queue.cc:237] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.908005 2748 raft_consensus.cc:348] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.908170 2748 raft_consensus.cc:374] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:46.908263 2748 raft_consensus.cc:729] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Initialized, Role: FOLLOWER
I20240502 14:16:46.908489 2748 consensus_queue.cc:260] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.908684 2748 raft_consensus.cc:388] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:46.908774 2748 raft_consensus.cc:482] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:46.908897 2748 raft_consensus.cc:3037] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:46.911598 2748 raft_consensus.cc:504] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.911810 2748 leader_election.cc:302] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2; no voters:
I20240502 14:16:46.911980 2748 leader_election.cc:288] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:46.912061 2751 raft_consensus.cc:2781] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:46.912333 2751 raft_consensus.cc:686] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:46.912573 2748 ts_tablet_manager.cc:1398] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 14:16:46.912425 2128 catalog_manager.cc:5484] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:46.912910 2748 tablet_bootstrap.cc:492] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap starting.
I20240502 14:16:46.912720 2751 consensus_queue.cc:237] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.915484 2748 tablet_bootstrap.cc:654] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:46.918850 2128 catalog_manager.cc:5484] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:46.920161 2748 tablet_bootstrap.cc:492] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2: No bootstrap required, opened a new log
I20240502 14:16:46.920317 2748 ts_tablet_manager.cc:1367] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:16:46.920898 2748 raft_consensus.cc:348] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.921033 2748 raft_consensus.cc:374] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:46.921110 2748 raft_consensus.cc:729] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Initialized, Role: FOLLOWER
I20240502 14:16:46.921315 2748 consensus_queue.cc:260] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.921458 2748 raft_consensus.cc:388] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:46.921547 2748 raft_consensus.cc:482] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:46.921630 2748 raft_consensus.cc:3037] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:46.923491 2748 raft_consensus.cc:504] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.923666 2748 leader_election.cc:302] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2; no voters:
I20240502 14:16:46.923816 2748 leader_election.cc:288] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:46.923880 2750 raft_consensus.cc:2781] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:46.924088 2750 raft_consensus.cc:686] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:46.924217 2748 ts_tablet_manager.cc:1398] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:16:46.924305 2750 consensus_queue.cc:237] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:46.926118 2128 catalog_manager.cc:5484] T 357f9a3dd69240c0aaff828e372210f0 P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:47.426754 2128 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45712:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:47.435185 2385 tablet_service.cc:1467] Processing CreateTablet for tablet 690f0d7773b449ed809305f1dba6596d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8f3630d9cb00452190d4cd6efb9d9e8d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:47.435243 2384 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad25c6e312f459fbc3ad0074843eadb (DEFAULT_TABLE table=rebalance_test_table_1 [id=8f3630d9cb00452190d4cd6efb9d9e8d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:47.435322 2383 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86efdd342142f88de569c37b55236d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8f3630d9cb00452190d4cd6efb9d9e8d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:47.435758 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690f0d7773b449ed809305f1dba6596d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:47.435984 2383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86efdd342142f88de569c37b55236d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:47.436188 2384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad25c6e312f459fbc3ad0074843eadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:47.442504 2782 tablet_bootstrap.cc:492] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a: Bootstrap starting.
I20240502 14:16:47.443899 2782 tablet_bootstrap.cc:654] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:47.445664 2782 tablet_bootstrap.cc:492] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a: No bootstrap required, opened a new log
I20240502 14:16:47.445792 2782 ts_tablet_manager.cc:1367] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:47.446537 2782 raft_consensus.cc:348] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.446687 2782 raft_consensus.cc:374] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:47.446787 2782 raft_consensus.cc:729] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Initialized, Role: FOLLOWER
I20240502 14:16:47.446996 2782 consensus_queue.cc:260] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.447183 2782 raft_consensus.cc:388] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:47.447257 2782 raft_consensus.cc:482] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:47.447372 2782 raft_consensus.cc:3037] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:47.449131 2782 raft_consensus.cc:504] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.449352 2782 leader_election.cc:302] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:47.449761 2782 leader_election.cc:288] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:47.449908 2784 raft_consensus.cc:2781] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:47.450486 2782 ts_tablet_manager.cc:1398] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:16:47.450481 2784 raft_consensus.cc:686] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Leader. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:47.450786 2782 tablet_bootstrap.cc:492] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: Bootstrap starting.
I20240502 14:16:47.450803 2784 consensus_queue.cc:237] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.452221 2782 tablet_bootstrap.cc:654] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:47.453306 2128 catalog_manager.cc:5484] T 8f86efdd342142f88de569c37b55236d P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66). New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:47.454054 2782 tablet_bootstrap.cc:492] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: No bootstrap required, opened a new log
I20240502 14:16:47.454203 2782 ts_tablet_manager.cc:1367] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:47.454907 2782 raft_consensus.cc:348] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.455112 2782 raft_consensus.cc:374] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:47.455200 2782 raft_consensus.cc:729] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Initialized, Role: FOLLOWER
I20240502 14:16:47.455439 2782 consensus_queue.cc:260] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.455595 2782 raft_consensus.cc:388] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:47.455682 2782 raft_consensus.cc:482] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:47.455803 2782 raft_consensus.cc:3037] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:47.458087 2782 raft_consensus.cc:504] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.458245 2782 leader_election.cc:302] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:47.458397 2782 leader_election.cc:288] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:47.458453 2784 raft_consensus.cc:2781] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:47.458622 2784 raft_consensus.cc:686] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Leader. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:47.458850 2782 ts_tablet_manager.cc:1398] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:16:47.458853 2784 consensus_queue.cc:237] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.459185 2782 tablet_bootstrap.cc:492] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: Bootstrap starting.
I20240502 14:16:47.460706 2782 tablet_bootstrap.cc:654] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:47.461174 2128 catalog_manager.cc:5484] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66). New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:47.462301 2782 tablet_bootstrap.cc:492] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: No bootstrap required, opened a new log
I20240502 14:16:47.462421 2782 ts_tablet_manager.cc:1367] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:47.463018 2782 raft_consensus.cc:348] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.463203 2782 raft_consensus.cc:374] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:47.463301 2782 raft_consensus.cc:729] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Initialized, Role: FOLLOWER
I20240502 14:16:47.463531 2782 consensus_queue.cc:260] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.463762 2782 raft_consensus.cc:388] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:47.463863 2782 raft_consensus.cc:482] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:47.463979 2782 raft_consensus.cc:3037] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:47.466140 2782 raft_consensus.cc:504] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.466346 2782 leader_election.cc:302] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:47.466513 2782 leader_election.cc:288] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:47.466609 2784 raft_consensus.cc:2781] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:47.466748 2784 raft_consensus.cc:686] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Leader. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:47.466969 2782 ts_tablet_manager.cc:1398] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:16:47.467005 2784 consensus_queue.cc:237] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:47.468711 2129 catalog_manager.cc:5484] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66). New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:47.961498 2129 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45712:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:47.969147 2252 tablet_service.cc:1467] Processing CreateTablet for tablet bcdaf9de71674e75b34c70da073a43f9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4fb5dfa71d3f49219204ca75fadf6844]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:47.969238 2251 tablet_service.cc:1467] Processing CreateTablet for tablet 507124e331034e6a93011064368cc3de (DEFAULT_TABLE table=rebalance_test_table_2 [id=4fb5dfa71d3f49219204ca75fadf6844]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:47.969521 2253 tablet_service.cc:1467] Processing CreateTablet for tablet e9a0102b02f144a78aa6ea6d46866647 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4fb5dfa71d3f49219204ca75fadf6844]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:47.969633 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdaf9de71674e75b34c70da073a43f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:47.969867 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507124e331034e6a93011064368cc3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:47.970181 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a0102b02f144a78aa6ea6d46866647. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:47.975557 2814 tablet_bootstrap.cc:492] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap starting.
I20240502 14:16:47.977171 2814 tablet_bootstrap.cc:654] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:47.978675 2814 tablet_bootstrap.cc:492] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: No bootstrap required, opened a new log
I20240502 14:16:47.978844 2814 ts_tablet_manager.cc:1367] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:47.979465 2814 raft_consensus.cc:348] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.979604 2814 raft_consensus.cc:374] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:47.979700 2814 raft_consensus.cc:729] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Initialized, Role: FOLLOWER
I20240502 14:16:47.979899 2814 consensus_queue.cc:260] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.980062 2814 raft_consensus.cc:388] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:47.980131 2814 raft_consensus.cc:482] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:47.980238 2814 raft_consensus.cc:3037] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:47.981889 2814 raft_consensus.cc:504] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.982100 2814 leader_election.cc:302] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2; no voters:
I20240502 14:16:47.982461 2814 leader_election.cc:288] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:47.982616 2817 raft_consensus.cc:2781] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:47.983310 2817 raft_consensus.cc:686] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:47.983551 2814 ts_tablet_manager.cc:1398] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:16:47.983569 2817 consensus_queue.cc:237] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.983785 2814 tablet_bootstrap.cc:492] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap starting.
I20240502 14:16:47.985340 2814 tablet_bootstrap.cc:654] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:47.985558 2129 catalog_manager.cc:5484] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:47.986960 2814 tablet_bootstrap.cc:492] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: No bootstrap required, opened a new log
I20240502 14:16:47.987136 2814 ts_tablet_manager.cc:1367] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:47.987761 2814 raft_consensus.cc:348] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.987941 2814 raft_consensus.cc:374] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:47.988026 2814 raft_consensus.cc:729] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Initialized, Role: FOLLOWER
I20240502 14:16:47.988246 2814 consensus_queue.cc:260] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.988418 2814 raft_consensus.cc:388] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:47.988519 2814 raft_consensus.cc:482] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:47.988611 2814 raft_consensus.cc:3037] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:47.990747 2814 raft_consensus.cc:504] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.990917 2814 leader_election.cc:302] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2; no voters:
I20240502 14:16:47.991061 2814 leader_election.cc:288] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:47.991132 2817 raft_consensus.cc:2781] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:47.991355 2817 raft_consensus.cc:686] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:47.991519 2814 ts_tablet_manager.cc:1398] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240502 14:16:47.991555 2817 consensus_queue.cc:237] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.991775 2814 tablet_bootstrap.cc:492] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap starting.
I20240502 14:16:47.993273 2814 tablet_bootstrap.cc:654] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:47.993583 2128 catalog_manager.cc:5484] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:47.995011 2814 tablet_bootstrap.cc:492] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2: No bootstrap required, opened a new log
I20240502 14:16:47.995155 2814 ts_tablet_manager.cc:1367] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:16:47.995800 2814 raft_consensus.cc:348] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.995937 2814 raft_consensus.cc:374] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:47.996052 2814 raft_consensus.cc:729] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Initialized, Role: FOLLOWER
I20240502 14:16:47.996276 2814 consensus_queue.cc:260] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.996476 2814 raft_consensus.cc:388] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:47.996567 2814 raft_consensus.cc:482] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:47.996688 2814 raft_consensus.cc:3037] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:47.998570 2814 raft_consensus.cc:504] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:47.998764 2814 leader_election.cc:302] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2; no voters:
I20240502 14:16:47.998903 2814 leader_election.cc:288] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:47.999001 2817 raft_consensus.cc:2781] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:47.999166 2817 raft_consensus.cc:686] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:47.999348 2814 ts_tablet_manager.cc:1398] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:16:47.999397 2817 consensus_queue.cc:237] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } }
I20240502 14:16:48.001181 2128 catalog_manager.cc:5484] T bcdaf9de71674e75b34c70da073a43f9 P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:48.504151 2128 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45712:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:16:48.511919 2383 tablet_service.cc:1467] Processing CreateTablet for tablet 70fa361fec5d44248dcb3229d6d446fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=8ba49d3afb734da3a589a9667fddad06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:48.512279 2384 tablet_service.cc:1467] Processing CreateTablet for tablet c6296ef8c8284176bfc4a263c82d7ff5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8ba49d3afb734da3a589a9667fddad06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:48.512413 2383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fa361fec5d44248dcb3229d6d446fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:48.512605 2384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6296ef8c8284176bfc4a263c82d7ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:48.511920 2385 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa80c09a76b4e81b80f3913fbcc4509 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8ba49d3afb734da3a589a9667fddad06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:16:48.516901 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa80c09a76b4e81b80f3913fbcc4509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:48.519440 2852 tablet_bootstrap.cc:492] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: Bootstrap starting.
I20240502 14:16:48.520810 2852 tablet_bootstrap.cc:654] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:48.522610 2852 tablet_bootstrap.cc:492] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: No bootstrap required, opened a new log
I20240502 14:16:48.522718 2852 ts_tablet_manager.cc:1367] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:16:48.523324 2852 raft_consensus.cc:348] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.523452 2852 raft_consensus.cc:374] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:48.523515 2852 raft_consensus.cc:729] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Initialized, Role: FOLLOWER
I20240502 14:16:48.523710 2852 consensus_queue.cc:260] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.523850 2852 raft_consensus.cc:388] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:48.523905 2852 raft_consensus.cc:482] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:48.523991 2852 raft_consensus.cc:3037] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:48.526026 2852 raft_consensus.cc:504] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.526212 2852 leader_election.cc:302] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:48.526566 2852 leader_election.cc:288] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:48.526706 2854 raft_consensus.cc:2781] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:48.527365 2852 ts_tablet_manager.cc:1398] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 14:16:48.527616 2852 tablet_bootstrap.cc:492] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a: Bootstrap starting.
I20240502 14:16:48.527627 2854 raft_consensus.cc:686] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Leader. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:48.527904 2854 consensus_queue.cc:237] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.529173 2852 tablet_bootstrap.cc:654] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:48.529996 2128 catalog_manager.cc:5484] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66). New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:48.530858 2852 tablet_bootstrap.cc:492] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a: No bootstrap required, opened a new log
I20240502 14:16:48.531004 2852 ts_tablet_manager.cc:1367] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:48.531678 2852 raft_consensus.cc:348] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.531872 2852 raft_consensus.cc:374] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:48.531960 2852 raft_consensus.cc:729] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Initialized, Role: FOLLOWER
I20240502 14:16:48.532176 2852 consensus_queue.cc:260] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.532333 2852 raft_consensus.cc:388] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:48.532433 2852 raft_consensus.cc:482] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:48.532526 2852 raft_consensus.cc:3037] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:48.534386 2852 raft_consensus.cc:504] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.534559 2852 leader_election.cc:302] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:48.534778 2852 leader_election.cc:288] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:48.534828 2854 raft_consensus.cc:2781] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:48.535003 2854 raft_consensus.cc:686] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Leader. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:48.535284 2852 ts_tablet_manager.cc:1398] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240502 14:16:48.535226 2854 consensus_queue.cc:237] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.535535 2852 tablet_bootstrap.cc:492] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: Bootstrap starting.
I20240502 14:16:48.537237 2852 tablet_bootstrap.cc:654] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:48.537564 2128 catalog_manager.cc:5484] T 70fa361fec5d44248dcb3229d6d446fb P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66). New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:48.538806 2852 tablet_bootstrap.cc:492] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: No bootstrap required, opened a new log
I20240502 14:16:48.538928 2852 ts_tablet_manager.cc:1367] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:16:48.539567 2852 raft_consensus.cc:348] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.539723 2852 raft_consensus.cc:374] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:48.539829 2852 raft_consensus.cc:729] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Initialized, Role: FOLLOWER
I20240502 14:16:48.540041 2852 consensus_queue.cc:260] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.540203 2852 raft_consensus.cc:388] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:48.540277 2852 raft_consensus.cc:482] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:48.540391 2852 raft_consensus.cc:3037] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:48.542289 2852 raft_consensus.cc:504] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.542491 2852 leader_election.cc:302] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:48.542641 2852 leader_election.cc:288] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:48.542721 2854 raft_consensus.cc:2781] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:48.542889 2854 raft_consensus.cc:686] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Leader. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:48.543108 2852 ts_tablet_manager.cc:1398] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:16:48.543133 2854 consensus_queue.cc:237] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } }
I20240502 14:16:48.544842 2128 catalog_manager.cc:5484] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66). New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } health_report { overall_health: HEALTHY } } }
I20240502 14:16:49.030445 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:38343
--local_ip_for_outbound_sockets=127.21.58.67
--tserver_master_addrs=127.21.58.126:42477
--webserver_port=43421
--webserver_interface=127.21.58.67
--builtin_ntp_servers=127.21.58.84:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:16:49.364317 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:49.364625 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:49.364761 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:49.372340 2884 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:49.372479 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:49.372627 2884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:16:49.381686 2884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:38343
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=43421
--tserver_master_addrs=127.21.58.126:42477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:49.383000 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:49.384770 2884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:49.396272 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:49.397039 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:49.397617 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:49.397904 2884 server_base.cc:1034] running on GCE node
I20240502 14:16:49.398415 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:49.399263 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:49.400454 2884 hybrid_clock.cc:648] HybridClock initialized: now 1714659409400411 us; error 40 us; skew 500 ppm
I20240502 14:16:49.400745 2884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:49.402746 2884 webserver.cc:460] Webserver started at http://127.21.58.67:43421/ using document root <none> and password file <none>
I20240502 14:16:49.403573 2884 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:49.403712 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:49.408596 2884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:16:49.411135 2898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:49.412305 2884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:49.412459 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "948dd66e071d426c95807b11c23599ce"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:49.413266 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:49.429390 2884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:49.430073 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:49.430316 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:49.431154 2884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:49.432559 2884 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:49.432634 2884 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:49.432760 2884 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:49.432816 2884 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:49.462543 2884 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:38343
I20240502 14:16:49.462594 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:38343 every 8 connection(s)
I20240502 14:16:49.463963 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:16:49.469229 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 2884
I20240502 14:16:49.469882 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:34509
--local_ip_for_outbound_sockets=127.21.58.68
--tserver_master_addrs=127.21.58.126:42477
--webserver_port=34585
--webserver_interface=127.21.58.68
--builtin_ntp_servers=127.21.58.84:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:49.479044 3011 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42477
I20240502 14:16:49.479357 3011 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:49.480185 3011 heartbeater.cc:511] Master 127.21.58.126:42477 requested a full tablet report, sending...
I20240502 14:16:49.481792 2128 ts_manager.cc:194] Re-registered known tserver with Master: 948dd66e071d426c95807b11c23599ce (127.21.58.67:38343)
I20240502 14:16:49.482623 2128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:58803
W20240502 14:16:49.822427 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:49.822726 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:49.822865 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:49.830511 3015 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:49.830685 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:49.830843 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:16:49.839906 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:34509
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=34585
--tserver_master_addrs=127.21.58.126:42477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:49.841281 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:49.843334 3015 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:49.855038 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:49.855738 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:49.856434 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:49.856717 3015 server_base.cc:1034] running on GCE node
I20240502 14:16:49.857275 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:49.858191 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:49.859383 3015 hybrid_clock.cc:648] HybridClock initialized: now 1714659409859334 us; error 38 us; skew 500 ppm
I20240502 14:16:49.859700 3015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:49.861914 3015 webserver.cc:460] Webserver started at http://127.21.58.68:34585/ using document root <none> and password file <none>
I20240502 14:16:49.862639 3015 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:49.862773 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:49.868010 3015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:49.870771 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:49.872289 3015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:16:49.872439 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "96bce754191b4a478796de231f93fb34"
format_stamp: "Formatted at 2024-05-02 14:16:42 on dist-test-slave-wtgm"
I20240502 14:16:49.873235 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:49.891945 3015 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:49.892715 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:49.892992 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:49.893887 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:49.895372 3015 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:49.895450 3015 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:49.895571 3015 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:49.895620 3015 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:49.925243 3015 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:34509
I20240502 14:16:49.925323 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:34509 every 8 connection(s)
I20240502 14:16:49.926687 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:16:49.930361 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 3015
I20240502 14:16:49.939249 3144 heartbeater.cc:347] Connected to a master server at 127.21.58.126:42477
I20240502 14:16:49.939533 3144 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:49.940305 3144 heartbeater.cc:511] Master 127.21.58.126:42477 requested a full tablet report, sending...
I20240502 14:16:49.941727 2128 ts_manager.cc:194] Re-registered known tserver with Master: 96bce754191b4a478796de231f93fb34 (127.21.58.68:34509)
I20240502 14:16:49.942499 2128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:44343
I20240502 14:16:50.413903 2252 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:50.433159 2946 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:50.436288 2385 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:16:50.453290 3078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:50.484840 3011 heartbeater.cc:503] Master 127.21.58.126:42477 was elected leader, sending a full tablet report...
I20240502 14:16:50.606725 2385 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:16:50.607719 2252 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:50.620004 2946 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:50.620563 3078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:50.782997 2385 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:16:50.785986 2252 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:16:50.790596 2946 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:50.806324 3078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:16:50.868008 2273 consensus_queue.cc:237] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:50.870613 3259 raft_consensus.cc:2932] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } } }
I20240502 14:16:50.876423 2126 catalog_manager.cc:5484] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:50.880882 2207 consensus_peers.cc:485] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 -> Peer 948dd66e071d426c95807b11c23599ce (127.21.58.67:38343): Couldn't send request to peer 948dd66e071d426c95807b11c23599ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 507124e331034e6a93011064368cc3de. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:50.896461 2405 consensus_queue.cc:237] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:50.898898 3272 raft_consensus.cc:2932] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 96bce754191b4a478796de231f93fb34 (127.21.58.68) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } } }
I20240502 14:16:50.905558 2126 catalog_manager.cc:5484] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index -1 to 3, NON_VOTER 96bce754191b4a478796de231f93fb34 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:50.909857 2340 consensus_peers.cc:485] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a -> Peer 96bce754191b4a478796de231f93fb34 (127.21.58.68:34509): Couldn't send request to peer 96bce754191b4a478796de231f93fb34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0aa80c09a76b4e81b80f3913fbcc4509. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:50.919449 2273 consensus_queue.cc:237] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:50.921027 3259 raft_consensus.cc:2932] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } } }
W20240502 14:16:50.921593 2207 consensus_peers.cc:485] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 -> Peer 948dd66e071d426c95807b11c23599ce (127.21.58.67:38343): Couldn't send request to peer 948dd66e071d426c95807b11c23599ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a9f0f44361a48799b2c978b1585bd1d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:50.925431 2126 catalog_manager.cc:5484] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:50.936805 2405 consensus_queue.cc:237] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: true } }
I20240502 14:16:50.938750 3272 raft_consensus.cc:2932] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: true } } }
I20240502 14:16:50.943893 2126 catalog_manager.cc:5484] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index -1 to 3, NON_VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) added. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:50.945413 2341 consensus_peers.cc:485] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a -> Peer 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409): Couldn't send request to peer 2c3e885d68564b61ba3401a80ee146e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 690f0d7773b449ed809305f1dba6596d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:50.945562 3144 heartbeater.cc:503] Master 127.21.58.126:42477 was elected leader, sending a full tablet report...
I20240502 14:16:50.955108 2273 consensus_queue.cc:237] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:50.956691 3259 raft_consensus.cc:2932] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 96bce754191b4a478796de231f93fb34 (127.21.58.68) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } } }
I20240502 14:16:50.962172 2126 catalog_manager.cc:5484] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index -1 to 3, NON_VOTER 96bce754191b4a478796de231f93fb34 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:50.963971 2207 consensus_peers.cc:485] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 -> Peer 96bce754191b4a478796de231f93fb34 (127.21.58.68:34509): Couldn't send request to peer 96bce754191b4a478796de231f93fb34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36ccbeda4b0f4efbbb8edc2b571e94dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:50.972898 2405 consensus_queue.cc:237] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:50.974547 3272 raft_consensus.cc:2932] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } } }
I20240502 14:16:50.978971 2126 catalog_manager.cc:5484] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:16:50.980342 2340 consensus_peers.cc:485] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a -> Peer 948dd66e071d426c95807b11c23599ce (127.21.58.67:38343): Couldn't send request to peer 948dd66e071d426c95807b11c23599ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ad25c6e312f459fbc3ad0074843eadb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:50.990170 2273 consensus_queue.cc:237] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:50.991760 3280 raft_consensus.cc:2932] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 96bce754191b4a478796de231f93fb34 (127.21.58.68) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } } }
W20240502 14:16:50.992309 2207 consensus_peers.cc:485] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 -> Peer 96bce754191b4a478796de231f93fb34 (127.21.58.68:34509): Couldn't send request to peer 96bce754191b4a478796de231f93fb34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9a0102b02f144a78aa6ea6d46866647. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:50.995903 2126 catalog_manager.cc:5484] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 96bce754191b4a478796de231f93fb34 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:51.005987 2405 consensus_queue.cc:237] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.007511 3271 raft_consensus.cc:2932] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } } }
W20240502 14:16:51.007905 2340 consensus_peers.cc:485] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a -> Peer 948dd66e071d426c95807b11c23599ce (127.21.58.67:38343): Couldn't send request to peer 948dd66e071d426c95807b11c23599ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6296ef8c8284176bfc4a263c82d7ff5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:51.011649 2126 catalog_manager.cc:5484] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index -1 to 2, NON_VOTER 948dd66e071d426c95807b11c23599ce (127.21.58.67) added. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:16:51.371246 3393 ts_tablet_manager.cc:902] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: Initiating tablet copy from peer 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:51.372912 3393 tablet_copy_client.cc:308] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:36889
I20240502 14:16:51.378319 2425 tablet_copy_service.cc:140] P 35dad2fd4bed4e71a86573ec6096908a: Received BeginTabletCopySession request for tablet 690f0d7773b449ed809305f1dba6596d from peer 2c3e885d68564b61ba3401a80ee146e2 ({username='slave'} at 127.21.58.65:50567)
I20240502 14:16:51.378572 2425 tablet_copy_service.cc:161] P 35dad2fd4bed4e71a86573ec6096908a: Beginning new tablet copy session on tablet 690f0d7773b449ed809305f1dba6596d from peer 2c3e885d68564b61ba3401a80ee146e2 at {username='slave'} at 127.21.58.65:50567: session id = 2c3e885d68564b61ba3401a80ee146e2-690f0d7773b449ed809305f1dba6596d
I20240502 14:16:51.381233 2425 tablet_copy_source_session.cc:205] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.382519 3398 ts_tablet_manager.cc:902] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: Initiating tablet copy from peer 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:51.383688 3393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690f0d7773b449ed809305f1dba6596d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.384438 3398 tablet_copy_client.cc:308] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:42409
I20240502 14:16:51.389801 3393 tablet_copy_client.cc:783] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.390105 3393 tablet_copy_client.cc:647] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.392082 2293 tablet_copy_service.cc:140] P 2c3e885d68564b61ba3401a80ee146e2: Received BeginTabletCopySession request for tablet 6a9f0f44361a48799b2c978b1585bd1d from peer 948dd66e071d426c95807b11c23599ce ({username='slave'} at 127.21.58.67:34693)
I20240502 14:16:51.392338 2293 tablet_copy_service.cc:161] P 2c3e885d68564b61ba3401a80ee146e2: Beginning new tablet copy session on tablet 6a9f0f44361a48799b2c978b1585bd1d from peer 948dd66e071d426c95807b11c23599ce at {username='slave'} at 127.21.58.67:34693: session id = 948dd66e071d426c95807b11c23599ce-6a9f0f44361a48799b2c978b1585bd1d
I20240502 14:16:51.392668 3393 tablet_copy_client.cc:523] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.396255 2293 tablet_copy_source_session.cc:205] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.396971 3393 tablet_bootstrap.cc:492] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap starting.
I20240502 14:16:51.401546 3398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9f0f44361a48799b2c978b1585bd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.406157 3393 tablet_bootstrap.cc:492] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.406657 3393 tablet_bootstrap.cc:492] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: Bootstrap complete.
I20240502 14:16:51.407016 3393 ts_tablet_manager.cc:1367] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 14:16:51.407913 3393 raft_consensus.cc:348] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: true } }
I20240502 14:16:51.408147 3393 raft_consensus.cc:729] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Initialized, Role: LEARNER
I20240502 14:16:51.408377 3393 consensus_queue.cc:260] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: true } }
I20240502 14:16:51.408957 3393 ts_tablet_manager.cc:1398] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:51.410089 2425 tablet_copy_service.cc:342] P 35dad2fd4bed4e71a86573ec6096908a: Request end of tablet copy session 2c3e885d68564b61ba3401a80ee146e2-690f0d7773b449ed809305f1dba6596d received from {username='slave'} at 127.21.58.65:50567
I20240502 14:16:51.410280 2425 tablet_copy_service.cc:433] P 35dad2fd4bed4e71a86573ec6096908a: ending tablet copy session 2c3e885d68564b61ba3401a80ee146e2-690f0d7773b449ed809305f1dba6596d on tablet 690f0d7773b449ed809305f1dba6596d with peer 2c3e885d68564b61ba3401a80ee146e2
I20240502 14:16:51.415678 3398 tablet_copy_client.cc:783] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.417137 3398 tablet_copy_client.cc:647] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.419062 3398 tablet_copy_client.cc:523] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.422046 3398 tablet_bootstrap.cc:492] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: Bootstrap starting.
I20240502 14:16:51.428299 3398 log.cc:824] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:51.431425 3398 tablet_bootstrap.cc:492] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.431826 3398 tablet_bootstrap.cc:492] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: Bootstrap complete.
I20240502 14:16:51.432176 3398 ts_tablet_manager.cc:1367] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: Time spent bootstrapping tablet: real 0.010s user 0.011s sys 0.000s
I20240502 14:16:51.437767 3398 raft_consensus.cc:348] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.438133 3398 raft_consensus.cc:729] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Initialized, Role: LEARNER
I20240502 14:16:51.438555 3398 consensus_queue.cc:260] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.439879 3398 ts_tablet_manager.cc:1398] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240502 14:16:51.440815 2293 tablet_copy_service.cc:342] P 2c3e885d68564b61ba3401a80ee146e2: Request end of tablet copy session 948dd66e071d426c95807b11c23599ce-6a9f0f44361a48799b2c978b1585bd1d received from {username='slave'} at 127.21.58.67:34693
I20240502 14:16:51.441000 2293 tablet_copy_service.cc:433] P 2c3e885d68564b61ba3401a80ee146e2: ending tablet copy session 948dd66e071d426c95807b11c23599ce-6a9f0f44361a48799b2c978b1585bd1d on tablet 6a9f0f44361a48799b2c978b1585bd1d with peer 948dd66e071d426c95807b11c23599ce
I20240502 14:16:51.454767 3407 ts_tablet_manager.cc:902] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: Initiating tablet copy from peer 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:51.456496 3407 tablet_copy_client.cc:308] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:36889
I20240502 14:16:51.462283 2425 tablet_copy_service.cc:140] P 35dad2fd4bed4e71a86573ec6096908a: Received BeginTabletCopySession request for tablet 0aa80c09a76b4e81b80f3913fbcc4509 from peer 96bce754191b4a478796de231f93fb34 ({username='slave'} at 127.21.58.68:60343)
I20240502 14:16:51.462466 2425 tablet_copy_service.cc:161] P 35dad2fd4bed4e71a86573ec6096908a: Beginning new tablet copy session on tablet 0aa80c09a76b4e81b80f3913fbcc4509 from peer 96bce754191b4a478796de231f93fb34 at {username='slave'} at 127.21.58.68:60343: session id = 96bce754191b4a478796de231f93fb34-0aa80c09a76b4e81b80f3913fbcc4509
I20240502 14:16:51.465416 2425 tablet_copy_source_session.cc:205] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.470093 3407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa80c09a76b4e81b80f3913fbcc4509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.473147 3398 ts_tablet_manager.cc:902] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: Initiating tablet copy from peer 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:51.474574 3398 tablet_copy_client.cc:308] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:42409
I20240502 14:16:51.475277 2293 tablet_copy_service.cc:140] P 2c3e885d68564b61ba3401a80ee146e2: Received BeginTabletCopySession request for tablet 507124e331034e6a93011064368cc3de from peer 948dd66e071d426c95807b11c23599ce ({username='slave'} at 127.21.58.67:34693)
I20240502 14:16:51.475462 2293 tablet_copy_service.cc:161] P 2c3e885d68564b61ba3401a80ee146e2: Beginning new tablet copy session on tablet 507124e331034e6a93011064368cc3de from peer 948dd66e071d426c95807b11c23599ce at {username='slave'} at 127.21.58.67:34693: session id = 948dd66e071d426c95807b11c23599ce-507124e331034e6a93011064368cc3de
I20240502 14:16:51.478149 2293 tablet_copy_source_session.cc:205] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.479488 3398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507124e331034e6a93011064368cc3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.479521 3417 ts_tablet_manager.cc:902] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: Initiating tablet copy from peer 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:51.483954 3398 tablet_copy_client.cc:783] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.484166 3398 tablet_copy_client.cc:647] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.484565 3417 tablet_copy_client.cc:308] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:42409
I20240502 14:16:51.485599 3398 tablet_copy_client.cc:523] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.485644 3407 tablet_copy_client.cc:783] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.485955 3407 tablet_copy_client.cc:647] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.487977 3407 tablet_copy_client.cc:523] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.488648 3398 tablet_bootstrap.cc:492] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: Bootstrap starting.
I20240502 14:16:51.491787 2293 tablet_copy_service.cc:140] P 2c3e885d68564b61ba3401a80ee146e2: Received BeginTabletCopySession request for tablet 36ccbeda4b0f4efbbb8edc2b571e94dd from peer 96bce754191b4a478796de231f93fb34 ({username='slave'} at 127.21.58.68:47507)
I20240502 14:16:51.491971 2293 tablet_copy_service.cc:161] P 2c3e885d68564b61ba3401a80ee146e2: Beginning new tablet copy session on tablet 36ccbeda4b0f4efbbb8edc2b571e94dd from peer 96bce754191b4a478796de231f93fb34 at {username='slave'} at 127.21.58.68:47507: session id = 96bce754191b4a478796de231f93fb34-36ccbeda4b0f4efbbb8edc2b571e94dd
I20240502 14:16:51.492267 3407 tablet_bootstrap.cc:492] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: Bootstrap starting.
I20240502 14:16:51.494036 3398 tablet_bootstrap.cc:492] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.494428 3398 tablet_bootstrap.cc:492] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: Bootstrap complete.
I20240502 14:16:51.494660 3398 ts_tablet_manager.cc:1367] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:16:51.494685 2293 tablet_copy_source_session.cc:205] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.495551 3398 raft_consensus.cc:348] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.495719 3398 raft_consensus.cc:729] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Initialized, Role: LEARNER
I20240502 14:16:51.495844 3398 consensus_queue.cc:260] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.495913 3417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ccbeda4b0f4efbbb8edc2b571e94dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.496268 3398 ts_tablet_manager.cc:1398] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:16:51.497313 2293 tablet_copy_service.cc:342] P 2c3e885d68564b61ba3401a80ee146e2: Request end of tablet copy session 948dd66e071d426c95807b11c23599ce-507124e331034e6a93011064368cc3de received from {username='slave'} at 127.21.58.67:34693
I20240502 14:16:51.497429 2293 tablet_copy_service.cc:433] P 2c3e885d68564b61ba3401a80ee146e2: ending tablet copy session 948dd66e071d426c95807b11c23599ce-507124e331034e6a93011064368cc3de on tablet 507124e331034e6a93011064368cc3de with peer 948dd66e071d426c95807b11c23599ce
I20240502 14:16:51.500634 3407 log.cc:824] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:51.501996 3417 tablet_copy_client.cc:783] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.502214 3417 tablet_copy_client.cc:647] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.503952 3417 tablet_copy_client.cc:523] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.507305 3417 tablet_bootstrap.cc:492] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: Bootstrap starting.
I20240502 14:16:51.511734 3407 tablet_bootstrap.cc:492] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.512250 3407 tablet_bootstrap.cc:492] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: Bootstrap complete.
I20240502 14:16:51.512677 3407 ts_tablet_manager.cc:1367] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.007s
I20240502 14:16:51.514467 3417 tablet_bootstrap.cc:492] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.514906 3417 tablet_bootstrap.cc:492] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: Bootstrap complete.
I20240502 14:16:51.515157 3417 ts_tablet_manager.cc:1367] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20240502 14:16:51.518306 3407 raft_consensus.cc:348] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:51.518330 3417 raft_consensus.cc:348] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:51.518693 3417 raft_consensus.cc:729] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96bce754191b4a478796de231f93fb34, State: Initialized, Role: LEARNER
I20240502 14:16:51.518699 3407 raft_consensus.cc:729] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96bce754191b4a478796de231f93fb34, State: Initialized, Role: LEARNER
I20240502 14:16:51.519162 3407 consensus_queue.cc:260] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:51.519169 3417 consensus_queue.cc:260] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:51.520774 3407 ts_tablet_manager.cc:1398] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 14:16:51.521827 2425 tablet_copy_service.cc:342] P 35dad2fd4bed4e71a86573ec6096908a: Request end of tablet copy session 96bce754191b4a478796de231f93fb34-0aa80c09a76b4e81b80f3913fbcc4509 received from {username='slave'} at 127.21.58.68:60343
I20240502 14:16:51.521968 2425 tablet_copy_service.cc:433] P 35dad2fd4bed4e71a86573ec6096908a: ending tablet copy session 96bce754191b4a478796de231f93fb34-0aa80c09a76b4e81b80f3913fbcc4509 on tablet 0aa80c09a76b4e81b80f3913fbcc4509 with peer 96bce754191b4a478796de231f93fb34
I20240502 14:16:51.522037 3417 ts_tablet_manager.cc:1398] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:16:51.522706 2293 tablet_copy_service.cc:342] P 2c3e885d68564b61ba3401a80ee146e2: Request end of tablet copy session 96bce754191b4a478796de231f93fb34-36ccbeda4b0f4efbbb8edc2b571e94dd received from {username='slave'} at 127.21.58.68:47507
I20240502 14:16:51.522861 2293 tablet_copy_service.cc:433] P 2c3e885d68564b61ba3401a80ee146e2: ending tablet copy session 96bce754191b4a478796de231f93fb34-36ccbeda4b0f4efbbb8edc2b571e94dd on tablet 36ccbeda4b0f4efbbb8edc2b571e94dd with peer 96bce754191b4a478796de231f93fb34
I20240502 14:16:51.537070 3398 ts_tablet_manager.cc:902] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: Initiating tablet copy from peer 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:51.538170 3398 tablet_copy_client.cc:308] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:36889
I20240502 14:16:51.543359 2425 tablet_copy_service.cc:140] P 35dad2fd4bed4e71a86573ec6096908a: Received BeginTabletCopySession request for tablet 8ad25c6e312f459fbc3ad0074843eadb from peer 948dd66e071d426c95807b11c23599ce ({username='slave'} at 127.21.58.67:52567)
I20240502 14:16:51.543483 2425 tablet_copy_service.cc:161] P 35dad2fd4bed4e71a86573ec6096908a: Beginning new tablet copy session on tablet 8ad25c6e312f459fbc3ad0074843eadb from peer 948dd66e071d426c95807b11c23599ce at {username='slave'} at 127.21.58.67:52567: session id = 948dd66e071d426c95807b11c23599ce-8ad25c6e312f459fbc3ad0074843eadb
I20240502 14:16:51.545372 2425 tablet_copy_source_session.cc:205] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.546417 3398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad25c6e312f459fbc3ad0074843eadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.550817 3398 tablet_copy_client.cc:783] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.551102 3398 tablet_copy_client.cc:647] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.552489 3398 tablet_copy_client.cc:523] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.554584 3417 ts_tablet_manager.cc:902] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: Initiating tablet copy from peer 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:51.554929 3398 tablet_bootstrap.cc:492] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: Bootstrap starting.
I20240502 14:16:51.558014 3417 tablet_copy_client.cc:308] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:42409
I20240502 14:16:51.558707 2293 tablet_copy_service.cc:140] P 2c3e885d68564b61ba3401a80ee146e2: Received BeginTabletCopySession request for tablet e9a0102b02f144a78aa6ea6d46866647 from peer 96bce754191b4a478796de231f93fb34 ({username='slave'} at 127.21.58.68:47507)
I20240502 14:16:51.558904 2293 tablet_copy_service.cc:161] P 2c3e885d68564b61ba3401a80ee146e2: Beginning new tablet copy session on tablet e9a0102b02f144a78aa6ea6d46866647 from peer 96bce754191b4a478796de231f93fb34 at {username='slave'} at 127.21.58.68:47507: session id = 96bce754191b4a478796de231f93fb34-e9a0102b02f144a78aa6ea6d46866647
I20240502 14:16:51.562595 2293 tablet_copy_source_session.cc:205] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.564114 3417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a0102b02f144a78aa6ea6d46866647. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.565632 3398 tablet_bootstrap.cc:492] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.566071 3398 tablet_bootstrap.cc:492] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: Bootstrap complete.
I20240502 14:16:51.566322 3398 ts_tablet_manager.cc:1367] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20240502 14:16:51.567065 3398 raft_consensus.cc:348] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.567344 3398 raft_consensus.cc:729] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Initialized, Role: LEARNER
I20240502 14:16:51.567580 3398 consensus_queue.cc:260] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.568138 3398 ts_tablet_manager.cc:1398] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:51.569308 2425 tablet_copy_service.cc:342] P 35dad2fd4bed4e71a86573ec6096908a: Request end of tablet copy session 948dd66e071d426c95807b11c23599ce-8ad25c6e312f459fbc3ad0074843eadb received from {username='slave'} at 127.21.58.67:52567
I20240502 14:16:51.569449 2425 tablet_copy_service.cc:433] P 35dad2fd4bed4e71a86573ec6096908a: ending tablet copy session 948dd66e071d426c95807b11c23599ce-8ad25c6e312f459fbc3ad0074843eadb on tablet 8ad25c6e312f459fbc3ad0074843eadb with peer 948dd66e071d426c95807b11c23599ce
I20240502 14:16:51.570070 3435 ts_tablet_manager.cc:902] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: Initiating tablet copy from peer 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:51.571065 3417 tablet_copy_client.cc:783] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.571367 3417 tablet_copy_client.cc:647] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.573169 3417 tablet_copy_client.cc:523] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.574221 3435 tablet_copy_client.cc:308] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:36889
I20240502 14:16:51.574963 2425 tablet_copy_service.cc:140] P 35dad2fd4bed4e71a86573ec6096908a: Received BeginTabletCopySession request for tablet c6296ef8c8284176bfc4a263c82d7ff5 from peer 948dd66e071d426c95807b11c23599ce ({username='slave'} at 127.21.58.67:52567)
I20240502 14:16:51.575187 2425 tablet_copy_service.cc:161] P 35dad2fd4bed4e71a86573ec6096908a: Beginning new tablet copy session on tablet c6296ef8c8284176bfc4a263c82d7ff5 from peer 948dd66e071d426c95807b11c23599ce at {username='slave'} at 127.21.58.67:52567: session id = 948dd66e071d426c95807b11c23599ce-c6296ef8c8284176bfc4a263c82d7ff5
I20240502 14:16:51.576694 3417 tablet_bootstrap.cc:492] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: Bootstrap starting.
I20240502 14:16:51.578194 2425 tablet_copy_source_session.cc:205] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:51.579373 3435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6296ef8c8284176bfc4a263c82d7ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:51.587177 3435 tablet_copy_client.cc:783] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:51.587563 3435 tablet_copy_client.cc:647] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:51.588464 3417 tablet_bootstrap.cc:492] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.588829 3417 tablet_bootstrap.cc:492] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: Bootstrap complete.
I20240502 14:16:51.589083 3417 ts_tablet_manager.cc:1367] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240502 14:16:51.589224 3435 tablet_copy_client.cc:523] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:51.589962 3417 raft_consensus.cc:348] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:51.590163 3417 raft_consensus.cc:729] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96bce754191b4a478796de231f93fb34, State: Initialized, Role: LEARNER
I20240502 14:16:51.590349 3417 consensus_queue.cc:260] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: true } }
I20240502 14:16:51.590798 3417 ts_tablet_manager.cc:1398] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:51.591524 2293 tablet_copy_service.cc:342] P 2c3e885d68564b61ba3401a80ee146e2: Request end of tablet copy session 96bce754191b4a478796de231f93fb34-e9a0102b02f144a78aa6ea6d46866647 received from {username='slave'} at 127.21.58.68:47507
I20240502 14:16:51.591629 2293 tablet_copy_service.cc:433] P 2c3e885d68564b61ba3401a80ee146e2: ending tablet copy session 96bce754191b4a478796de231f93fb34-e9a0102b02f144a78aa6ea6d46866647 on tablet e9a0102b02f144a78aa6ea6d46866647 with peer 96bce754191b4a478796de231f93fb34
I20240502 14:16:51.592299 3435 tablet_bootstrap.cc:492] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: Bootstrap starting.
I20240502 14:16:51.599401 3435 tablet_bootstrap.cc:492] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:51.599735 3435 tablet_bootstrap.cc:492] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: Bootstrap complete.
I20240502 14:16:51.599928 3435 ts_tablet_manager.cc:1367] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:16:51.600530 3435 raft_consensus.cc:348] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.600754 3435 raft_consensus.cc:729] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Initialized, Role: LEARNER
I20240502 14:16:51.600950 3435 consensus_queue.cc:260] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: true } }
I20240502 14:16:51.601585 3435 ts_tablet_manager.cc:1398] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:51.602293 2425 tablet_copy_service.cc:342] P 35dad2fd4bed4e71a86573ec6096908a: Request end of tablet copy session 948dd66e071d426c95807b11c23599ce-c6296ef8c8284176bfc4a263c82d7ff5 received from {username='slave'} at 127.21.58.67:52567
I20240502 14:16:51.602391 2425 tablet_copy_service.cc:433] P 35dad2fd4bed4e71a86573ec6096908a: ending tablet copy session 948dd66e071d426c95807b11c23599ce-c6296ef8c8284176bfc4a263c82d7ff5 on tablet c6296ef8c8284176bfc4a263c82d7ff5 with peer 948dd66e071d426c95807b11c23599ce
I20240502 14:16:51.836480 2273 raft_consensus.cc:1203] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 14:16:51.901592 3098 raft_consensus.cc:1203] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 14:16:51.912282 3098 raft_consensus.cc:1203] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 14:16:51.945772 2966 raft_consensus.cc:1203] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:52.003394 2966 raft_consensus.cc:1203] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:52.008191 2966 raft_consensus.cc:1203] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:52.156024 3098 raft_consensus.cc:1203] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:52.184535 2966 raft_consensus.cc:1203] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:52.259693 3486 raft_consensus.cc:1050] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: attempting to promote NON_VOTER 2c3e885d68564b61ba3401a80ee146e2 to VOTER
I20240502 14:16:52.260488 3486 consensus_queue.cc:237] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } }
I20240502 14:16:52.262192 2273 raft_consensus.cc:1261] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEARNER]: Refusing update from remote peer 35dad2fd4bed4e71a86573ec6096908a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:52.262768 3487 consensus_queue.cc:1035] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:52.265234 3486 raft_consensus.cc:2932] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } } }
I20240502 14:16:52.265794 2273 raft_consensus.cc:2932] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } } }
I20240502 14:16:52.269373 2126 catalog_manager.cc:5484] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index 3 to 4, 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.343956 3272 raft_consensus.cc:1050] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: attempting to promote NON_VOTER 96bce754191b4a478796de231f93fb34 to VOTER
I20240502 14:16:52.344524 3272 consensus_queue.cc:237] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:52.346307 3098 raft_consensus.cc:1261] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Refusing update from remote peer 35dad2fd4bed4e71a86573ec6096908a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:52.346794 3486 consensus_queue.cc:1035] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:52.349798 3272 raft_consensus.cc:2932] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:52.350484 3098 raft_consensus.cc:2932] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:52.355134 2129 catalog_manager.cc:5484] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index 3 to 4, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.426226 3486 raft_consensus.cc:1050] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: attempting to promote NON_VOTER 948dd66e071d426c95807b11c23599ce to VOTER
I20240502 14:16:52.426824 3486 consensus_queue.cc:237] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:52.428915 2966 raft_consensus.cc:1261] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Refusing update from remote peer 35dad2fd4bed4e71a86573ec6096908a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:52.429476 3486 consensus_queue.cc:1035] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Connected to new peer: Peer: permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:52.440702 3487 raft_consensus.cc:2932] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.441501 2966 raft_consensus.cc:2932] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.444792 2405 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0aa80c09a76b4e81b80f3913fbcc4509"
dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53718
I20240502 14:16:52.445029 2405 raft_consensus.cc:595] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.446565 2126 catalog_manager.cc:5484] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.472108 2405 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "690f0d7773b449ed809305f1dba6596d"
dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53736
I20240502 14:16:52.472293 2405 raft_consensus.cc:595] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.478142 3514 raft_consensus.cc:1050] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: attempting to promote NON_VOTER 96bce754191b4a478796de231f93fb34 to VOTER
I20240502 14:16:52.479353 3514 consensus_queue.cc:237] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:52.481412 3098 raft_consensus.cc:1261] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Refusing update from remote peer 2c3e885d68564b61ba3401a80ee146e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:52.481921 3513 consensus_queue.cc:1035] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:52.484423 3514 raft_consensus.cc:2932] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:52.485030 3098 raft_consensus.cc:2932] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:52.488859 2129 catalog_manager.cc:5484] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 reported cstate change: config changed from index 3 to 4, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:52.496923 2405 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ad25c6e312f459fbc3ad0074843eadb"
dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53744
I20240502 14:16:52.497084 2405 raft_consensus.cc:595] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.514688 3259 raft_consensus.cc:1050] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: attempting to promote NON_VOTER 948dd66e071d426c95807b11c23599ce to VOTER
I20240502 14:16:52.515278 3259 consensus_queue.cc:237] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:52.517066 2966 raft_consensus.cc:1261] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Refusing update from remote peer 2c3e885d68564b61ba3401a80ee146e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:52.517670 3513 consensus_queue.cc:1035] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:52.520136 3259 raft_consensus.cc:2932] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.520748 2966 raft_consensus.cc:2932] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.524114 2273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36ccbeda4b0f4efbbb8edc2b571e94dd"
dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47684
I20240502 14:16:52.524350 2273 raft_consensus.cc:595] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.524382 2126 catalog_manager.cc:5484] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.536403 3513 raft_consensus.cc:1050] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: attempting to promote NON_VOTER 948dd66e071d426c95807b11c23599ce to VOTER
I20240502 14:16:52.536826 3513 consensus_queue.cc:237] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:52.538321 2966 raft_consensus.cc:1261] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Refusing update from remote peer 2c3e885d68564b61ba3401a80ee146e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:52.538717 3259 consensus_queue.cc:1035] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:52.540933 3513 raft_consensus.cc:2932] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.541515 2966 raft_consensus.cc:2932] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.545584 2129 catalog_manager.cc:5484] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.685665 3514 raft_consensus.cc:1050] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: attempting to promote NON_VOTER 96bce754191b4a478796de231f93fb34 to VOTER
I20240502 14:16:52.686264 3514 consensus_queue.cc:237] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:52.687940 3098 raft_consensus.cc:1261] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 1 LEARNER]: Refusing update from remote peer 2c3e885d68564b61ba3401a80ee146e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:52.688385 3513 consensus_queue.cc:1035] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:52.690593 3514 raft_consensus.cc:2932] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:52.691229 3098 raft_consensus.cc:2932] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:52.694934 2126 catalog_manager.cc:5484] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index 2 to 3, 96bce754191b4a478796de231f93fb34 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.739391 3272 raft_consensus.cc:979] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: : Instructing follower 96bce754191b4a478796de231f93fb34 to start an election
I20240502 14:16:52.739537 3272 raft_consensus.cc:1067] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Signalling peer 96bce754191b4a478796de231f93fb34 to start an election
I20240502 14:16:52.740541 3098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0aa80c09a76b4e81b80f3913fbcc4509"
dest_uuid: "96bce754191b4a478796de231f93fb34"
from {username='slave'} at 127.21.58.66:39853
I20240502 14:16:52.740844 3098 raft_consensus.cc:482] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:52.740995 3098 raft_consensus.cc:3037] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:52.743450 3098 raft_consensus.cc:504] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:52.745047 3098 leader_election.cc:288] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [CANDIDATE]: Term 2 election: Requested vote from peers 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:52.745615 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa80c09a76b4e81b80f3913fbcc4509" candidate_uuid: "96bce754191b4a478796de231f93fb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
I20240502 14:16:52.745874 2405 raft_consensus.cc:3032] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:52.745990 2405 raft_consensus.cc:729] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:52.746284 2405 consensus_queue.cc:260] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:52.746604 2405 raft_consensus.cc:3037] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:52.748975 3272 raft_consensus.cc:1050] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: attempting to promote NON_VOTER 948dd66e071d426c95807b11c23599ce to VOTER
I20240502 14:16:52.749256 2405 raft_consensus.cc:2454] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96bce754191b4a478796de231f93fb34 in term 2.
I20240502 14:16:52.749584 3272 consensus_queue.cc:237] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:52.749744 3032 leader_election.cc:302] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a, 96bce754191b4a478796de231f93fb34; no voters:
I20240502 14:16:52.750092 3426 raft_consensus.cc:2781] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:52.750469 3426 raft_consensus.cc:686] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 2 LEADER]: Becoming Leader. State: Replica: 96bce754191b4a478796de231f93fb34, State: Running, Role: LEADER
I20240502 14:16:52.750921 3426 consensus_queue.cc:237] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:52.751224 2966 raft_consensus.cc:1261] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 1 LEARNER]: Refusing update from remote peer 35dad2fd4bed4e71a86573ec6096908a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:52.751713 3486 consensus_queue.cc:1035] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [LEADER]: Connected to new peer: Peer: permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:52.753806 2129 catalog_manager.cc:5484] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 reported cstate change: term changed from 1 to 2, leader changed from 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) to 96bce754191b4a478796de231f93fb34 (127.21.58.68). New cstate: current_term: 2 leader_uuid: "96bce754191b4a478796de231f93fb34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.755061 3272 raft_consensus.cc:2932] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.755683 2966 raft_consensus.cc:2932] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:52.756497 2273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9a0102b02f144a78aa6ea6d46866647"
dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47696
I20240502 14:16:52.756716 2273 raft_consensus.cc:595] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.763674 2129 catalog_manager.cc:5484] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a reported cstate change: config changed from index 2 to 3, 948dd66e071d426c95807b11c23599ce (127.21.58.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "35dad2fd4bed4e71a86573ec6096908a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:52.769114 3272 raft_consensus.cc:979] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: : Instructing follower 2c3e885d68564b61ba3401a80ee146e2 to start an election
I20240502 14:16:52.769264 3272 raft_consensus.cc:1067] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Signalling peer 2c3e885d68564b61ba3401a80ee146e2 to start an election
I20240502 14:16:52.769876 2273 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "690f0d7773b449ed809305f1dba6596d"
dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
from {username='slave'} at 127.21.58.66:55315
I20240502 14:16:52.770085 2273 raft_consensus.cc:482] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:52.770182 2273 raft_consensus.cc:3037] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:52.773017 2273 raft_consensus.cc:504] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } }
I20240502 14:16:52.774312 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690f0d7773b449ed809305f1dba6596d" candidate_uuid: "2c3e885d68564b61ba3401a80ee146e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
I20240502 14:16:52.774482 2405 raft_consensus.cc:3032] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:52.774561 2405 raft_consensus.cc:729] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:52.774772 2405 consensus_queue.cc:260] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } }
I20240502 14:16:52.774981 2405 raft_consensus.cc:3037] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:52.777603 2405 raft_consensus.cc:2454] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3e885d68564b61ba3401a80ee146e2 in term 2.
I20240502 14:16:52.778162 2207 leader_election.cc:302] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2, 35dad2fd4bed4e71a86573ec6096908a; no voters:
I20240502 14:16:52.778510 3513 raft_consensus.cc:2781] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:52.778620 3513 raft_consensus.cc:686] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 LEADER]: Becoming Leader. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:52.778873 3513 consensus_queue.cc:237] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } }
I20240502 14:16:52.780469 2273 leader_election.cc:288] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [CANDIDATE]: Term 2 election: Requested vote from peers 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:52.784003 2126 catalog_manager.cc:5484] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: term changed from 1 to 2, leader changed from 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) to 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65). New cstate: current_term: 2 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:16:52.784566 2115 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 0aa80c09a76b4e81b80f3913fbcc4509 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 96bce754191b4a478796de231f93fb34 (127.21.58.68:34509) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:16:52.814711 2273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "507124e331034e6a93011064368cc3de"
dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47714
I20240502 14:16:52.814888 2273 raft_consensus.cc:595] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.840220 2405 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ad25c6e312f459fbc3ad0074843eadb"
dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53778
I20240502 14:16:52.840377 2405 raft_consensus.cc:595] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.856801 2273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a9f0f44361a48799b2c978b1585bd1d"
dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47748
I20240502 14:16:52.856977 2273 raft_consensus.cc:595] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:52.872694 2273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36ccbeda4b0f4efbbb8edc2b571e94dd"
dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47774
I20240502 14:16:52.872833 2273 raft_consensus.cc:595] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:53.034623 3486 raft_consensus.cc:979] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: : Instructing follower 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.034809 3486 raft_consensus.cc:1067] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Signalling peer 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.035465 2966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ad25c6e312f459fbc3ad0074843eadb"
dest_uuid: "948dd66e071d426c95807b11c23599ce"
from {username='slave'} at 127.21.58.66:36467
I20240502 14:16:53.035712 2966 raft_consensus.cc:482] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:53.035929 2966 raft_consensus.cc:3037] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.037868 2966 raft_consensus.cc:504] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.039175 2966 leader_election.cc:288] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Requested vote from peers 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:53.039603 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad25c6e312f459fbc3ad0074843eadb" candidate_uuid: "948dd66e071d426c95807b11c23599ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
I20240502 14:16:53.039781 2405 raft_consensus.cc:3032] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:53.039877 2405 raft_consensus.cc:729] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:53.040108 2405 consensus_queue.cc:260] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.040362 2405 raft_consensus.cc:3037] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.042143 2405 raft_consensus.cc:2454] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948dd66e071d426c95807b11c23599ce in term 2.
I20240502 14:16:53.042611 2901 leader_election.cc:302] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a, 948dd66e071d426c95807b11c23599ce; no voters:
I20240502 14:16:53.042927 3406 raft_consensus.cc:2781] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:53.043373 3406 raft_consensus.cc:686] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Becoming Leader. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Running, Role: LEADER
I20240502 14:16:53.043843 3406 consensus_queue.cc:237] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.047065 2126 catalog_manager.cc:5484] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce reported cstate change: term changed from 1 to 2, leader changed from 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) to 948dd66e071d426c95807b11c23599ce (127.21.58.67). New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.069496 3514 raft_consensus.cc:979] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: : Instructing follower 96bce754191b4a478796de231f93fb34 to start an election
I20240502 14:16:53.069672 3514 raft_consensus.cc:1067] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Signalling peer 96bce754191b4a478796de231f93fb34 to start an election
I20240502 14:16:53.070353 3098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "36ccbeda4b0f4efbbb8edc2b571e94dd"
dest_uuid: "96bce754191b4a478796de231f93fb34"
from {username='slave'} at 127.21.58.65:41357
I20240502 14:16:53.070531 3098 raft_consensus.cc:482] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:53.070638 3098 raft_consensus.cc:3037] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.072558 3098 raft_consensus.cc:504] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.073107 3098 leader_election.cc:288] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [CANDIDATE]: Term 2 election: Requested vote from peers 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:53.073647 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ccbeda4b0f4efbbb8edc2b571e94dd" candidate_uuid: "96bce754191b4a478796de231f93fb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
I20240502 14:16:53.073933 2273 raft_consensus.cc:3032] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:53.074034 2273 raft_consensus.cc:729] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:53.074285 2273 consensus_queue.cc:260] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.074582 2273 raft_consensus.cc:3037] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.076665 2273 raft_consensus.cc:2454] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96bce754191b4a478796de231f93fb34 in term 2.
I20240502 14:16:53.077126 3033 leader_election.cc:302] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2, 96bce754191b4a478796de231f93fb34; no voters:
I20240502 14:16:53.077373 3426 raft_consensus.cc:2781] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:53.077541 3426 raft_consensus.cc:686] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 2 LEADER]: Becoming Leader. State: Replica: 96bce754191b4a478796de231f93fb34, State: Running, Role: LEADER
I20240502 14:16:53.077795 3426 consensus_queue.cc:237] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.079835 2126 catalog_manager.cc:5484] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 reported cstate change: term changed from 1 to 2, leader changed from 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) to 96bce754191b4a478796de231f93fb34 (127.21.58.68). New cstate: current_term: 2 leader_uuid: "96bce754191b4a478796de231f93fb34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.084349 3513 raft_consensus.cc:979] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: : Instructing follower 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.084522 3513 raft_consensus.cc:1067] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Signalling peer 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.085155 2966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "507124e331034e6a93011064368cc3de"
dest_uuid: "948dd66e071d426c95807b11c23599ce"
from {username='slave'} at 127.21.58.65:38583
I20240502 14:16:53.085378 2966 raft_consensus.cc:482] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:53.085492 2966 raft_consensus.cc:3037] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.088096 2966 raft_consensus.cc:504] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.088722 2966 leader_election.cc:288] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Requested vote from peers 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:53.089263 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507124e331034e6a93011064368cc3de" candidate_uuid: "948dd66e071d426c95807b11c23599ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
I20240502 14:16:53.089427 2273 raft_consensus.cc:3032] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:53.089516 2273 raft_consensus.cc:729] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:53.089766 2273 consensus_queue.cc:260] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.090123 2273 raft_consensus.cc:3037] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.092376 2273 raft_consensus.cc:2454] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948dd66e071d426c95807b11c23599ce in term 2.
I20240502 14:16:53.092772 2902 leader_election.cc:302] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2, 948dd66e071d426c95807b11c23599ce; no voters:
I20240502 14:16:53.093175 3406 raft_consensus.cc:2781] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:53.093333 3406 raft_consensus.cc:686] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Becoming Leader. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Running, Role: LEADER
I20240502 14:16:53.093581 3406 consensus_queue.cc:237] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.094863 2405 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6296ef8c8284176bfc4a263c82d7ff5"
dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53794
I20240502 14:16:53.095041 2405 raft_consensus.cc:595] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:53.095749 2126 catalog_manager.cc:5484] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce reported cstate change: term changed from 1 to 2, leader changed from 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) to 948dd66e071d426c95807b11c23599ce (127.21.58.67). New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.106871 3513 raft_consensus.cc:979] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: : Instructing follower 96bce754191b4a478796de231f93fb34 to start an election
I20240502 14:16:53.107043 3513 raft_consensus.cc:1067] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Signalling peer 96bce754191b4a478796de231f93fb34 to start an election
I20240502 14:16:53.107650 3098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9a0102b02f144a78aa6ea6d46866647"
dest_uuid: "96bce754191b4a478796de231f93fb34"
from {username='slave'} at 127.21.58.65:41357
I20240502 14:16:53.107884 3098 raft_consensus.cc:482] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:53.108028 3098 raft_consensus.cc:3037] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.110365 3098 raft_consensus.cc:504] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.110868 3098 leader_election.cc:288] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [CANDIDATE]: Term 2 election: Requested vote from peers 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:53.111342 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a0102b02f144a78aa6ea6d46866647" candidate_uuid: "96bce754191b4a478796de231f93fb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
I20240502 14:16:53.111523 2273 raft_consensus.cc:3032] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:53.111647 2273 raft_consensus.cc:729] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:53.111886 2273 consensus_queue.cc:260] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.112196 2273 raft_consensus.cc:3037] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.114621 2273 raft_consensus.cc:2454] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96bce754191b4a478796de231f93fb34 in term 2.
I20240502 14:16:53.114948 3033 leader_election.cc:302] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2, 96bce754191b4a478796de231f93fb34; no voters:
I20240502 14:16:53.115280 3426 raft_consensus.cc:2781] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:53.115453 3426 raft_consensus.cc:686] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 2 LEADER]: Becoming Leader. State: Replica: 96bce754191b4a478796de231f93fb34, State: Running, Role: LEADER
I20240502 14:16:53.115566 2273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9a0102b02f144a78aa6ea6d46866647"
dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47788
I20240502 14:16:53.115725 2273 raft_consensus.cc:595] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Received request to transfer leadership
I20240502 14:16:53.115690 3426 consensus_queue.cc:237] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.115828 2273 raft_consensus.cc:603] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 14:16:53.118309 2129 catalog_manager.cc:5484] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 reported cstate change: term changed from 1 to 2, leader changed from 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) to 96bce754191b4a478796de231f93fb34 (127.21.58.68). New cstate: current_term: 2 leader_uuid: "96bce754191b4a478796de231f93fb34" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.146739 3514 raft_consensus.cc:979] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: : Instructing follower 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.146888 3514 raft_consensus.cc:1067] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Signalling peer 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.147578 2966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a9f0f44361a48799b2c978b1585bd1d"
dest_uuid: "948dd66e071d426c95807b11c23599ce"
from {username='slave'} at 127.21.58.65:38583
I20240502 14:16:53.147754 2966 raft_consensus.cc:482] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:53.147868 2966 raft_consensus.cc:3037] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.149777 2966 raft_consensus.cc:504] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.150367 2966 leader_election.cc:288] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Requested vote from peers 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409)
I20240502 14:16:53.150614 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9f0f44361a48799b2c978b1585bd1d" candidate_uuid: "948dd66e071d426c95807b11c23599ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c3e885d68564b61ba3401a80ee146e2"
I20240502 14:16:53.150796 2273 raft_consensus.cc:3032] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:53.150864 2273 raft_consensus.cc:729] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c3e885d68564b61ba3401a80ee146e2, State: Running, Role: LEADER
I20240502 14:16:53.151135 2273 consensus_queue.cc:260] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.151386 2273 raft_consensus.cc:3037] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.153621 2273 raft_consensus.cc:2454] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948dd66e071d426c95807b11c23599ce in term 2.
I20240502 14:16:53.153995 2902 leader_election.cc:302] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c3e885d68564b61ba3401a80ee146e2, 948dd66e071d426c95807b11c23599ce; no voters:
I20240502 14:16:53.154302 3406 raft_consensus.cc:2781] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:53.154429 3406 raft_consensus.cc:686] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Becoming Leader. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Running, Role: LEADER
I20240502 14:16:53.154688 3406 consensus_queue.cc:237] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.156577 2129 catalog_manager.cc:5484] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce reported cstate change: term changed from 1 to 2, leader changed from 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) to 948dd66e071d426c95807b11c23599ce (127.21.58.67). New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.315778 3487 raft_consensus.cc:979] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: : Instructing follower 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.315862 2405 raft_consensus.cc:1261] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Refusing update from remote peer 96bce754191b4a478796de231f93fb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:16:53.316028 3486 raft_consensus.cc:1067] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Signalling peer 948dd66e071d426c95807b11c23599ce to start an election
I20240502 14:16:53.316634 2966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6296ef8c8284176bfc4a263c82d7ff5"
dest_uuid: "948dd66e071d426c95807b11c23599ce"
from {username='slave'} at 127.21.58.66:36467
I20240502 14:16:53.316646 3664 consensus_queue.cc:1035] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:53.316888 2966 raft_consensus.cc:482] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:53.316998 2966 raft_consensus.cc:3037] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.320813 2966 raft_consensus.cc:504] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.321250 2966 leader_election.cc:288] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Requested vote from peers 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889)
I20240502 14:16:53.325071 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6296ef8c8284176bfc4a263c82d7ff5" candidate_uuid: "948dd66e071d426c95807b11c23599ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "35dad2fd4bed4e71a86573ec6096908a"
I20240502 14:16:53.325246 2405 raft_consensus.cc:3032] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:53.325325 2405 raft_consensus.cc:729] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35dad2fd4bed4e71a86573ec6096908a, State: Running, Role: LEADER
I20240502 14:16:53.325569 2405 consensus_queue.cc:260] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.325964 2405 raft_consensus.cc:3037] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:53.329370 2405 raft_consensus.cc:2454] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948dd66e071d426c95807b11c23599ce in term 2.
I20240502 14:16:53.329793 2901 leader_election.cc:302] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35dad2fd4bed4e71a86573ec6096908a, 948dd66e071d426c95807b11c23599ce; no voters:
I20240502 14:16:53.330093 3406 raft_consensus.cc:2781] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:53.330248 3406 raft_consensus.cc:686] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Becoming Leader. State: Replica: 948dd66e071d426c95807b11c23599ce, State: Running, Role: LEADER
I20240502 14:16:53.330467 3406 consensus_queue.cc:237] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.332577 2129 catalog_manager.cc:5484] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce reported cstate change: term changed from 1 to 2, leader changed from 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) to 948dd66e071d426c95807b11c23599ce (127.21.58.67). New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.350391 2405 raft_consensus.cc:1261] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Refusing update from remote peer 2c3e885d68564b61ba3401a80ee146e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:16:53.350873 3513 consensus_queue.cc:1035] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:53.450376 2273 consensus_queue.cc:237] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } }
I20240502 14:16:53.451179 3514 raft_consensus.cc:2932] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } } }
I20240502 14:16:53.453580 2116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 690f0d7773b449ed809305f1dba6596d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.454761 2126 catalog_manager.cc:5484] T 690f0d7773b449ed809305f1dba6596d P 2c3e885d68564b61ba3401a80ee146e2 reported cstate change: config changed from index 4 to 6, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "2c3e885d68564b61ba3401a80ee146e2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.458110 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet 690f0d7773b449ed809305f1dba6596d with delete_type TABLET_DATA_TOMBSTONED (TS 35dad2fd4bed4e71a86573ec6096908a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59596
I20240502 14:16:53.458808 3814 tablet_replica.cc:330] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: stopping tablet replica
I20240502 14:16:53.459213 3814 raft_consensus.cc:2229] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.459432 3814 raft_consensus.cc:2258] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.460693 3814 ts_tablet_manager.cc:1875] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.465121 3814 ts_tablet_manager.cc:1888] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:16:53.465247 3814 log.cc:1196] T 690f0d7773b449ed809305f1dba6596d P 35dad2fd4bed4e71a86573ec6096908a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/690f0d7773b449ed809305f1dba6596d
I20240502 14:16:53.465991 2115 catalog_manager.cc:4830] TS 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889): tablet 690f0d7773b449ed809305f1dba6596d (table rebalance_test_table_1 [id=8f3630d9cb00452190d4cd6efb9d9e8d]) successfully deleted
I20240502 14:16:53.470629 3098 consensus_queue.cc:237] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.471431 3664 raft_consensus.cc:2932] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:53.474223 2115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0aa80c09a76b4e81b80f3913fbcc4509 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.475409 2126 catalog_manager.cc:5484] T 0aa80c09a76b4e81b80f3913fbcc4509 P 96bce754191b4a478796de231f93fb34 reported cstate change: config changed from index 4 to 6, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "96bce754191b4a478796de231f93fb34" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.478298 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa80c09a76b4e81b80f3913fbcc4509 with delete_type TABLET_DATA_TOMBSTONED (TS 35dad2fd4bed4e71a86573ec6096908a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59596
I20240502 14:16:53.478627 3814 tablet_replica.cc:330] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: stopping tablet replica
I20240502 14:16:53.478806 3814 raft_consensus.cc:2229] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.478974 3814 raft_consensus.cc:2258] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.479578 3814 ts_tablet_manager.cc:1875] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.483476 3814 ts_tablet_manager.cc:1888] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:16:53.483582 3814 log.cc:1196] T 0aa80c09a76b4e81b80f3913fbcc4509 P 35dad2fd4bed4e71a86573ec6096908a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/0aa80c09a76b4e81b80f3913fbcc4509
I20240502 14:16:53.484122 2115 catalog_manager.cc:4830] TS 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889): tablet 0aa80c09a76b4e81b80f3913fbcc4509 (table rebalance_test_table_3 [id=8ba49d3afb734da3a589a9667fddad06]) successfully deleted
I20240502 14:16:53.533872 2273 raft_consensus.cc:1261] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Refusing update from remote peer 96bce754191b4a478796de231f93fb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:53.534469 3664 consensus_queue.cc:1035] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:53.590153 2405 raft_consensus.cc:1261] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Refusing update from remote peer 948dd66e071d426c95807b11c23599ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:53.590924 3718 consensus_queue.cc:1035] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:53.631487 2273 raft_consensus.cc:1261] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Refusing update from remote peer 96bce754191b4a478796de231f93fb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:16:53.632004 3426 consensus_queue.cc:1035] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:53.635483 2272 raft_consensus.cc:1261] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Refusing update from remote peer 948dd66e071d426c95807b11c23599ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:53.636073 3406 consensus_queue.cc:1035] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:53.653681 2966 consensus_queue.cc:237] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.654472 3406 raft_consensus.cc:2932] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:53.658321 2115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ad25c6e312f459fbc3ad0074843eadb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.659279 2126 catalog_manager.cc:5484] T 8ad25c6e312f459fbc3ad0074843eadb P 948dd66e071d426c95807b11c23599ce reported cstate change: config changed from index 3 to 5, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.663199 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad25c6e312f459fbc3ad0074843eadb with delete_type TABLET_DATA_TOMBSTONED (TS 35dad2fd4bed4e71a86573ec6096908a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59596
I20240502 14:16:53.663512 3814 tablet_replica.cc:330] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: stopping tablet replica
I20240502 14:16:53.663686 3814 raft_consensus.cc:2229] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.663849 3814 raft_consensus.cc:2258] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.664582 3814 ts_tablet_manager.cc:1875] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.669855 3814 ts_tablet_manager.cc:1888] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:16:53.669960 3814 log.cc:1196] T 8ad25c6e312f459fbc3ad0074843eadb P 35dad2fd4bed4e71a86573ec6096908a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ad25c6e312f459fbc3ad0074843eadb
I20240502 14:16:53.670656 2115 catalog_manager.cc:4830] TS 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889): tablet 8ad25c6e312f459fbc3ad0074843eadb (table rebalance_test_table_1 [id=8f3630d9cb00452190d4cd6efb9d9e8d]) successfully deleted
I20240502 14:16:53.688630 3098 consensus_queue.cc:237] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.689369 3664 raft_consensus.cc:2932] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:53.692780 2115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 36ccbeda4b0f4efbbb8edc2b571e94dd with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.693560 2126 catalog_manager.cc:5484] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 96bce754191b4a478796de231f93fb34 reported cstate change: config changed from index 4 to 6, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "96bce754191b4a478796de231f93fb34" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.696789 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ccbeda4b0f4efbbb8edc2b571e94dd with delete_type TABLET_DATA_TOMBSTONED (TS 2c3e885d68564b61ba3401a80ee146e2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47376
I20240502 14:16:53.697758 3848 tablet_replica.cc:330] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: stopping tablet replica
I20240502 14:16:53.698199 3848 raft_consensus.cc:2229] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.698532 3848 raft_consensus.cc:2258] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.699858 3848 ts_tablet_manager.cc:1875] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.705936 3848 ts_tablet_manager.cc:1888] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:16:53.706051 3848 log.cc:1196] T 36ccbeda4b0f4efbbb8edc2b571e94dd P 2c3e885d68564b61ba3401a80ee146e2: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/36ccbeda4b0f4efbbb8edc2b571e94dd
I20240502 14:16:53.706799 2116 catalog_manager.cc:4830] TS 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409): tablet 36ccbeda4b0f4efbbb8edc2b571e94dd (table rebalance_test_table_0 [id=77bc6f9f1fc541c69d3501c0664659d8]) successfully deleted
I20240502 14:16:53.718894 2272 raft_consensus.cc:1261] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Refusing update from remote peer 948dd66e071d426c95807b11c23599ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:53.719373 3718 consensus_queue.cc:1035] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3e885d68564b61ba3401a80ee146e2" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 42409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:53.731317 2966 consensus_queue.cc:237] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.731981 3718 raft_consensus.cc:2932] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:53.735813 2115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a9f0f44361a48799b2c978b1585bd1d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.736243 2126 catalog_manager.cc:5484] T 6a9f0f44361a48799b2c978b1585bd1d P 948dd66e071d426c95807b11c23599ce reported cstate change: config changed from index 3 to 5, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.739475 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9f0f44361a48799b2c978b1585bd1d with delete_type TABLET_DATA_TOMBSTONED (TS 2c3e885d68564b61ba3401a80ee146e2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47376
I20240502 14:16:53.739786 3848 tablet_replica.cc:330] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: stopping tablet replica
I20240502 14:16:53.740003 3848 raft_consensus.cc:2229] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.740186 3848 raft_consensus.cc:2258] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.740994 3848 ts_tablet_manager.cc:1875] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.743427 2966 consensus_queue.cc:237] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.744186 3718 raft_consensus.cc:2932] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:53.746104 3848 ts_tablet_manager.cc:1888] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:16:53.746213 3848 log.cc:1196] T 6a9f0f44361a48799b2c978b1585bd1d P 2c3e885d68564b61ba3401a80ee146e2: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a9f0f44361a48799b2c978b1585bd1d
I20240502 14:16:53.746886 2116 catalog_manager.cc:4830] TS 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409): tablet 6a9f0f44361a48799b2c978b1585bd1d (table rebalance_test_table_0 [id=77bc6f9f1fc541c69d3501c0664659d8]) successfully deleted
I20240502 14:16:53.747871 2115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 507124e331034e6a93011064368cc3de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.748787 2126 catalog_manager.cc:5484] T 507124e331034e6a93011064368cc3de P 948dd66e071d426c95807b11c23599ce reported cstate change: config changed from index 3 to 5, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.752157 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 507124e331034e6a93011064368cc3de with delete_type TABLET_DATA_TOMBSTONED (TS 2c3e885d68564b61ba3401a80ee146e2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47376
I20240502 14:16:53.752485 3848 tablet_replica.cc:330] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: stopping tablet replica
I20240502 14:16:53.752655 3848 raft_consensus.cc:2229] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.752828 3848 raft_consensus.cc:2258] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.753556 3848 ts_tablet_manager.cc:1875] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.755410 3098 consensus_queue.cc:237] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } }
I20240502 14:16:53.756399 3426 raft_consensus.cc:2932] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } } }
I20240502 14:16:53.759817 2115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9a0102b02f144a78aa6ea6d46866647 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.760627 2126 catalog_manager.cc:5484] T e9a0102b02f144a78aa6ea6d46866647 P 96bce754191b4a478796de231f93fb34 reported cstate change: config changed from index 3 to 5, VOTER 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "96bce754191b4a478796de231f93fb34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96bce754191b4a478796de231f93fb34" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 34509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.763974 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a0102b02f144a78aa6ea6d46866647 with delete_type TABLET_DATA_TOMBSTONED (TS 2c3e885d68564b61ba3401a80ee146e2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47376
I20240502 14:16:53.776103 3848 ts_tablet_manager.cc:1888] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:16:53.776196 3848 log.cc:1196] T 507124e331034e6a93011064368cc3de P 2c3e885d68564b61ba3401a80ee146e2: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/507124e331034e6a93011064368cc3de
I20240502 14:16:53.776847 3848 tablet_replica.cc:330] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: stopping tablet replica
I20240502 14:16:53.777012 3848 raft_consensus.cc:2229] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.777205 3848 raft_consensus.cc:2258] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.777307 2116 catalog_manager.cc:4830] TS 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409): tablet 507124e331034e6a93011064368cc3de (table rebalance_test_table_2 [id=4fb5dfa71d3f49219204ca75fadf6844]) successfully deleted
I20240502 14:16:53.778103 3848 ts_tablet_manager.cc:1875] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.783561 3848 ts_tablet_manager.cc:1888] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:16:53.783681 3848 log.cc:1196] T e9a0102b02f144a78aa6ea6d46866647 P 2c3e885d68564b61ba3401a80ee146e2: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/e9a0102b02f144a78aa6ea6d46866647
I20240502 14:16:53.784379 2116 catalog_manager.cc:4830] TS 2c3e885d68564b61ba3401a80ee146e2 (127.21.58.65:42409): tablet e9a0102b02f144a78aa6ea6d46866647 (table rebalance_test_table_2 [id=4fb5dfa71d3f49219204ca75fadf6844]) successfully deleted
I20240502 14:16:53.884120 2405 raft_consensus.cc:1261] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Refusing update from remote peer 948dd66e071d426c95807b11c23599ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:53.884691 3718 consensus_queue.cc:1035] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dad2fd4bed4e71a86573ec6096908a" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:53.978374 2966 consensus_queue.cc:237] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } }
I20240502 14:16:53.979231 3718 raft_consensus.cc:2932] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } } }
I20240502 14:16:53.981701 2115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6296ef8c8284176bfc4a263c82d7ff5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:16:53.982934 2129 catalog_manager.cc:5484] T c6296ef8c8284176bfc4a263c82d7ff5 P 948dd66e071d426c95807b11c23599ce reported cstate change: config changed from index 3 to 5, VOTER 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "948dd66e071d426c95807b11c23599ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "948dd66e071d426c95807b11c23599ce" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 38343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:16:53.985891 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet c6296ef8c8284176bfc4a263c82d7ff5 with delete_type TABLET_DATA_TOMBSTONED (TS 35dad2fd4bed4e71a86573ec6096908a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59596
I20240502 14:16:53.986191 3814 tablet_replica.cc:330] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: stopping tablet replica
I20240502 14:16:53.986352 3814 raft_consensus.cc:2229] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:53.986506 3814 raft_consensus.cc:2258] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:53.987160 3814 ts_tablet_manager.cc:1875] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:53.991236 3814 ts_tablet_manager.cc:1888] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:16:53.991322 3814 log.cc:1196] T c6296ef8c8284176bfc4a263c82d7ff5 P 35dad2fd4bed4e71a86573ec6096908a: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/c6296ef8c8284176bfc4a263c82d7ff5
I20240502 14:16:53.991842 2115 catalog_manager.cc:4830] TS 35dad2fd4bed4e71a86573ec6096908a (127.21.58.66:36889): tablet c6296ef8c8284176bfc4a263c82d7ff5 (table rebalance_test_table_3 [id=8ba49d3afb734da3a589a9667fddad06]) successfully deleted
I20240502 14:16:54.101042 2252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.104967 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.114902 3078 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.134006 2946 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:16:54.262383 2252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.270233 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.270226 2946 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:16:54.273525 3078 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.553413 2252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.558630 2385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:16:54.559069 2946 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:16:54.567557 3078 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8e34e188582247178f4c1687d1bdeb0c | 127.21.58.126:42477 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:45249 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2c3e885d68564b61ba3401a80ee146e2 | 127.21.58.65:42409 | HEALTHY | <none> | 3 | 0
35dad2fd4bed4e71a86573ec6096908a | 127.21.58.66:36889 | HEALTHY | <none> | 3 | 0
948dd66e071d426c95807b11c23599ce | 127.21.58.67:38343 | HEALTHY | <none> | 4 | 0
96bce754191b4a478796de231f93fb34 | 127.21.58.68:34509 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.21.58.65 | experimental | 127.21.58.65:42409
local_ip_for_outbound_sockets | 127.21.58.66 | experimental | 127.21.58.66:36889
local_ip_for_outbound_sockets | 127.21.58.67 | experimental | 127.21.58.67:38343
local_ip_for_outbound_sockets | 127.21.58.68 | experimental | 127.21.58.68:34509
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.58.65:42409
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.58.66:36889
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.58.67:38343
server_dump_info_path | /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.21.58.68:34509
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.21.58.84:45249 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:16:54.613267 21737 log_verifier.cc:126] Checking tablet 0aa80c09a76b4e81b80f3913fbcc4509
I20240502 14:16:54.622090 21737 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0aa80c09a76b4e81b80f3913fbcc4509
I20240502 14:16:54.622201 21737 log_verifier.cc:126] Checking tablet 357f9a3dd69240c0aaff828e372210f0
I20240502 14:16:54.629601 21737 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 357f9a3dd69240c0aaff828e372210f0
I20240502 14:16:54.629685 21737 log_verifier.cc:126] Checking tablet 36ccbeda4b0f4efbbb8edc2b571e94dd
I20240502 14:16:54.637764 21737 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 36ccbeda4b0f4efbbb8edc2b571e94dd
I20240502 14:16:54.637856 21737 log_verifier.cc:126] Checking tablet 507124e331034e6a93011064368cc3de
I20240502 14:16:54.645854 21737 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 507124e331034e6a93011064368cc3de
I20240502 14:16:54.645938 21737 log_verifier.cc:126] Checking tablet 690f0d7773b449ed809305f1dba6596d
I20240502 14:16:54.654189 21737 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 690f0d7773b449ed809305f1dba6596d
I20240502 14:16:54.654270 21737 log_verifier.cc:126] Checking tablet 6a9f0f44361a48799b2c978b1585bd1d
I20240502 14:16:54.661882 21737 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 6a9f0f44361a48799b2c978b1585bd1d
I20240502 14:16:54.661962 21737 log_verifier.cc:126] Checking tablet 70fa361fec5d44248dcb3229d6d446fb
I20240502 14:16:54.669211 21737 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 70fa361fec5d44248dcb3229d6d446fb
I20240502 14:16:54.669288 21737 log_verifier.cc:126] Checking tablet 821c64a4d89e4dd88d21397253348a10
I20240502 14:16:54.676604 21737 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 821c64a4d89e4dd88d21397253348a10
I20240502 14:16:54.676683 21737 log_verifier.cc:126] Checking tablet 8ad25c6e312f459fbc3ad0074843eadb
I20240502 14:16:54.684576 21737 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 8ad25c6e312f459fbc3ad0074843eadb
I20240502 14:16:54.684656 21737 log_verifier.cc:126] Checking tablet 8f86efdd342142f88de569c37b55236d
I20240502 14:16:54.691744 21737 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8f86efdd342142f88de569c37b55236d
I20240502 14:16:54.691829 21737 log_verifier.cc:126] Checking tablet bcdaf9de71674e75b34c70da073a43f9
I20240502 14:16:54.699064 21737 log_verifier.cc:177] Verified matching terms for 2 ops in tablet bcdaf9de71674e75b34c70da073a43f9
I20240502 14:16:54.699188 21737 log_verifier.cc:126] Checking tablet c6296ef8c8284176bfc4a263c82d7ff5
I20240502 14:16:54.706665 21737 log_verifier.cc:177] Verified matching terms for 5 ops in tablet c6296ef8c8284176bfc4a263c82d7ff5
I20240502 14:16:54.706743 21737 log_verifier.cc:126] Checking tablet e9a0102b02f144a78aa6ea6d46866647
I20240502 14:16:54.714598 21737 log_verifier.cc:177] Verified matching terms for 5 ops in tablet e9a0102b02f144a78aa6ea6d46866647
I20240502 14:16:54.715703 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 2190
I20240502 14:16:54.716979 2313 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:54.847295 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 2190
W20240502 14:16:54.869638 2116 connection.cc:535] server connection from 127.21.58.65:56511 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:54.869994 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 2323
I20240502 14:16:54.870990 2446 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:54.991549 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 2323
W20240502 14:16:55.014134 2113 connection.cc:535] server connection from 127.21.58.66:52691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:55.014362 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 2884
I20240502 14:16:55.020268 3006 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:55.125414 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 2884
I20240502 14:16:55.145884 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 3015
I20240502 14:16:55.151561 3139 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:55.257082 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 3015
I20240502 14:16:55.277526 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 2098
I20240502 14:16:55.278620 2159 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:55.404922 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 2098
2024-05-02T14:16:55Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (15174 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (15174 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240502 14:16:55.463879 21737 test_util.cc:268] Using random seed: 1096718232
I20240502 14:16:55.465727 21737 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:16:55.465803 21737 ts_itest-base.cc:113] --------------
I20240502 14:16:55.465839 21737 ts_itest-base.cc:114] 6 tablet servers
I20240502 14:16:55.465886 21737 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:16:55.465921 21737 ts_itest-base.cc:116] --------------
2024-05-02T14:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:16:55Z Disabled control of system clock
I20240502 14:16:55.543016 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:37789
--webserver_interface=127.21.58.126
--webserver_port=0
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.58.126:37789
--location_mapping_cmd=/tmp/dist-test-task3cgK4A/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 14:16:55.880548 4000 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 14:16:55.880877 4000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:55.880946 4000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:55.881019 4000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:55.888446 4000 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:55.888557 4000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:16:55.888628 4000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:55.888674 4000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:16:55.888738 4000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:16:55.898015 4000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task3cgK4A/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.21.58.126:37789
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:37789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.58.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:55.899309 4000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:55.901042 4000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:55.911512 4005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:55.912145 4006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:55.912346 4008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:55.913040 4000 server_base.cc:1034] running on GCE node
I20240502 14:16:55.913682 4000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:55.914914 4000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:55.916123 4000 hybrid_clock.cc:648] HybridClock initialized: now 1714659415916058 us; error 44 us; skew 500 ppm
I20240502 14:16:55.916396 4000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:55.918444 4000 webserver.cc:460] Webserver started at http://127.21.58.126:42247/ using document root <none> and password file <none>
I20240502 14:16:55.919147 4000 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:55.919270 4000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:55.919541 4000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:55.923691 4000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a1c87fb826554567958ecbea47c66582"
format_stamp: "Formatted at 2024-05-02 14:16:55 on dist-test-slave-wtgm"
I20240502 14:16:55.924436 4000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a1c87fb826554567958ecbea47c66582"
format_stamp: "Formatted at 2024-05-02 14:16:55 on dist-test-slave-wtgm"
I20240502 14:16:55.929190 4000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:16:55.932165 4014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:55.933311 4000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:55.933467 4000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a1c87fb826554567958ecbea47c66582"
format_stamp: "Formatted at 2024-05-02 14:16:55 on dist-test-slave-wtgm"
I20240502 14:16:55.933624 4000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:55.956322 4000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:55.957103 4000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:55.957389 4000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:55.974637 4000 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.126:37789
I20240502 14:16:55.974656 4065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.126:37789 every 8 connection(s)
I20240502 14:16:55.976068 4000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:16:55.979435 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 4000
I20240502 14:16:55.979614 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:16:55.979848 4066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:55.992566 4066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582: Bootstrap starting.
I20240502 14:16:55.995589 4066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582: Neither blocks nor log segments found. Creating new log.
I20240502 14:16:55.996541 4066 log.cc:824] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582: Log is configured to *not* fsync() on all Append() calls
I20240502 14:16:55.998883 4066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582: No bootstrap required, opened a new log
I20240502 14:16:56.005342 4066 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c87fb826554567958ecbea47c66582" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 37789 } }
I20240502 14:16:56.005637 4066 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:16:56.005709 4066 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c87fb826554567958ecbea47c66582, State: Initialized, Role: FOLLOWER
I20240502 14:16:56.006327 4066 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c87fb826554567958ecbea47c66582" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 37789 } }
I20240502 14:16:56.006515 4066 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:16:56.006635 4066 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:16:56.006800 4066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:16:56.008894 4066 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c87fb826554567958ecbea47c66582" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 37789 } }
I20240502 14:16:56.009351 4066 leader_election.cc:302] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1c87fb826554567958ecbea47c66582; no voters:
I20240502 14:16:56.010030 4066 leader_election.cc:288] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:16:56.010152 4071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:16:56.010885 4071 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [term 1 LEADER]: Becoming Leader. State: Replica: a1c87fb826554567958ecbea47c66582, State: Running, Role: LEADER
I20240502 14:16:56.011421 4071 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c87fb826554567958ecbea47c66582" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 37789 } }
I20240502 14:16:56.012068 4066 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:16:56.014140 4073 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1c87fb826554567958ecbea47c66582. Latest consensus state: current_term: 1 leader_uuid: "a1c87fb826554567958ecbea47c66582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c87fb826554567958ecbea47c66582" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 37789 } } }
I20240502 14:16:56.014165 4072 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1c87fb826554567958ecbea47c66582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c87fb826554567958ecbea47c66582" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 37789 } } }
I20240502 14:16:56.014510 4073 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [sys.catalog]: This master's current role is: LEADER
I20240502 14:16:56.014511 4072 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582 [sys.catalog]: This master's current role is: LEADER
I20240502 14:16:56.022001 4077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:16:56.026744 4077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:16:56.035109 4077 catalog_manager.cc:1347] Generated new cluster ID: 247217be1d6b4226ade4980a1a2b9404
I20240502 14:16:56.035212 4077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:16:56.043684 4077 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:16:56.044759 4077 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:16:56.064596 4077 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a1c87fb826554567958ecbea47c66582: Generated new TSK 0
I20240502 14:16:56.065444 4077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:16:56.081964 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.65:0
--local_ip_for_outbound_sockets=127.21.58.65
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:16:56.412590 4090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:56.412863 4090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:56.412945 4090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:56.419919 4090 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:56.420017 4090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:56.420125 4090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.65
I20240502 14:16:56.428349 4090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.65
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:56.429513 4090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:56.431092 4090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:56.442337 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:56.442946 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:56.443482 4098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:56.443819 4090 server_base.cc:1034] running on GCE node
I20240502 14:16:56.444321 4090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:56.445158 4090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:56.446339 4090 hybrid_clock.cc:648] HybridClock initialized: now 1714659416446299 us; error 34 us; skew 500 ppm
I20240502 14:16:56.446620 4090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:56.448707 4090 webserver.cc:460] Webserver started at http://127.21.58.65:45701/ using document root <none> and password file <none>
I20240502 14:16:56.449436 4090 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:56.449551 4090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:56.449849 4090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:56.452280 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "79041fe14ee64478b7601e9b2230ccff"
format_stamp: "Formatted at 2024-05-02 14:16:56 on dist-test-slave-wtgm"
I20240502 14:16:56.453011 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "79041fe14ee64478b7601e9b2230ccff"
format_stamp: "Formatted at 2024-05-02 14:16:56 on dist-test-slave-wtgm"
I20240502 14:16:56.457721 4090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 14:16:56.460881 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:56.462054 4090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:56.462173 4090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "79041fe14ee64478b7601e9b2230ccff"
format_stamp: "Formatted at 2024-05-02 14:16:56 on dist-test-slave-wtgm"
I20240502 14:16:56.462337 4090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:56.485724 4090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:56.486479 4090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:56.486701 4090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:56.487640 4090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:56.489087 4090 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:56.489171 4090 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:56.489249 4090 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:56.489305 4090 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:56.519753 4090 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.65:41803
I20240502 14:16:56.519795 4216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.65:41803 every 8 connection(s)
I20240502 14:16:56.520900 4090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:16:56.530450 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 4090
I20240502 14:16:56.530675 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:16:56.533025 4217 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:16:56.533349 4217 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:56.534062 4217 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:16:56.534318 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.66:0
--local_ip_for_outbound_sockets=127.21.58.66
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:56.580900 4030 ts_manager.cc:194] Registered new tserver with Master: 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:16:56.582588 4030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.65:49611
W20240502 14:16:56.862385 4222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:56.862651 4222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:56.862748 4222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:56.870232 4222 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:56.870337 4222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:56.870466 4222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.66
I20240502 14:16:56.879235 4222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.66
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:56.880409 4222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:56.882025 4222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:56.893333 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:56.893478 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:56.894028 4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:56.894652 4222 server_base.cc:1034] running on GCE node
I20240502 14:16:56.895251 4222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:56.896103 4222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:56.897293 4222 hybrid_clock.cc:648] HybridClock initialized: now 1714659416897252 us; error 32 us; skew 500 ppm
I20240502 14:16:56.897588 4222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:56.899636 4222 webserver.cc:460] Webserver started at http://127.21.58.66:41651/ using document root <none> and password file <none>
I20240502 14:16:56.900305 4222 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:56.900414 4222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:56.900697 4222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:56.903134 4222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "751046160dd4466eb3f9ffea4128cdc5"
format_stamp: "Formatted at 2024-05-02 14:16:56 on dist-test-slave-wtgm"
I20240502 14:16:56.903923 4222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "751046160dd4466eb3f9ffea4128cdc5"
format_stamp: "Formatted at 2024-05-02 14:16:56 on dist-test-slave-wtgm"
I20240502 14:16:56.911600 4222 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240502 14:16:56.915803 4236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:56.917388 4222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:56.917586 4222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "751046160dd4466eb3f9ffea4128cdc5"
format_stamp: "Formatted at 2024-05-02 14:16:56 on dist-test-slave-wtgm"
I20240502 14:16:56.917773 4222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:56.936573 4222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:56.937304 4222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:56.937577 4222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:56.938431 4222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:56.939805 4222 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:56.939904 4222 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:56.940001 4222 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:56.940064 4222 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:56.968937 4222 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.66:32867
I20240502 14:16:56.968992 4348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.66:32867 every 8 connection(s)
I20240502 14:16:56.970237 4222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:16:56.976724 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 4222
I20240502 14:16:56.976943 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:16:56.980378 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:0
--local_ip_for_outbound_sockets=127.21.58.67
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:56.984545 4349 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:16:56.984897 4349 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:56.985926 4349 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:16:57.050065 4030 ts_manager.cc:194] Registered new tserver with Master: 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:16:57.050992 4030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.66:34717
W20240502 14:16:57.324720 4352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:57.324972 4352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:57.325085 4352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:57.332324 4352 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:57.332450 4352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:57.332595 4352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:16:57.341240 4352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:57.342507 4352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:57.344187 4352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:57.354678 4359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:57.355002 4358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:57.355552 4361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:57.356050 4352 server_base.cc:1034] running on GCE node
I20240502 14:16:57.356559 4352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:57.357398 4352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:57.358585 4352 hybrid_clock.cc:648] HybridClock initialized: now 1714659417358535 us; error 42 us; skew 500 ppm
I20240502 14:16:57.358872 4352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:57.360924 4352 webserver.cc:460] Webserver started at http://127.21.58.67:43109/ using document root <none> and password file <none>
I20240502 14:16:57.361593 4352 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:57.361721 4352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:57.362013 4352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:57.364495 4352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8f899bd3e9a148f5923c0277de1b3551"
format_stamp: "Formatted at 2024-05-02 14:16:57 on dist-test-slave-wtgm"
I20240502 14:16:57.365263 4352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8f899bd3e9a148f5923c0277de1b3551"
format_stamp: "Formatted at 2024-05-02 14:16:57 on dist-test-slave-wtgm"
I20240502 14:16:57.370038 4352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 14:16:57.373265 4367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:57.374471 4352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 14:16:57.374610 4352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8f899bd3e9a148f5923c0277de1b3551"
format_stamp: "Formatted at 2024-05-02 14:16:57 on dist-test-slave-wtgm"
I20240502 14:16:57.374804 4352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:57.408056 4352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:57.408879 4352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:57.409140 4352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:57.410037 4352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:57.411571 4352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:57.411713 4352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:57.411854 4352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:57.411931 4352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:57.445027 4352 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:39879
I20240502 14:16:57.445092 4480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:39879 every 8 connection(s)
I20240502 14:16:57.446256 4352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:16:57.454891 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 4352
I20240502 14:16:57.455094 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:16:57.458221 4481 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:16:57.458479 4481 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:57.458518 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:0
--local_ip_for_outbound_sockets=127.21.58.68
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:57.459318 4481 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:16:57.513536 4030 ts_manager.cc:194] Registered new tserver with Master: 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:16:57.514751 4030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:38507
I20240502 14:16:57.585503 4217 heartbeater.cc:503] Master 127.21.58.126:37789 was elected leader, sending a full tablet report...
W20240502 14:16:57.785604 4484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:57.785864 4484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:57.785976 4484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:57.792981 4484 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:57.793113 4484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:57.793241 4484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:16:57.801668 4484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:57.802876 4484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:57.804462 4484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:57.815271 4490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:57.815817 4491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:57.816377 4493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:57.816573 4484 server_base.cc:1034] running on GCE node
I20240502 14:16:57.817163 4484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:57.817994 4484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:57.819154 4484 hybrid_clock.cc:648] HybridClock initialized: now 1714659417819112 us; error 33 us; skew 500 ppm
I20240502 14:16:57.819415 4484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:57.821344 4484 webserver.cc:460] Webserver started at http://127.21.58.68:35571/ using document root <none> and password file <none>
I20240502 14:16:57.822057 4484 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:57.822158 4484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:57.822429 4484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:57.824836 4484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "42b2fb8410fe4df48fb3d243f813c322"
format_stamp: "Formatted at 2024-05-02 14:16:57 on dist-test-slave-wtgm"
I20240502 14:16:57.825536 4484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "42b2fb8410fe4df48fb3d243f813c322"
format_stamp: "Formatted at 2024-05-02 14:16:57 on dist-test-slave-wtgm"
I20240502 14:16:57.830269 4484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 14:16:57.833222 4499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:57.834321 4484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:16:57.834448 4484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "42b2fb8410fe4df48fb3d243f813c322"
format_stamp: "Formatted at 2024-05-02 14:16:57 on dist-test-slave-wtgm"
I20240502 14:16:57.834611 4484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:57.849779 4484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:57.850533 4484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:57.850795 4484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:57.851744 4484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:57.853147 4484 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:57.853231 4484 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:57.853333 4484 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:57.853402 4484 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:57.883342 4484 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:41023
I20240502 14:16:57.883401 4611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:41023 every 8 connection(s)
I20240502 14:16:57.884500 4484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:16:57.890754 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 4484
I20240502 14:16:57.890908 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:16:57.894040 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:0
--local_ip_for_outbound_sockets=127.21.58.69
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:57.896270 4612 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:16:57.896562 4612 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:57.897447 4612 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:16:57.940683 4030 ts_manager.cc:194] Registered new tserver with Master: 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023)
I20240502 14:16:57.941922 4030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:51813
I20240502 14:16:58.053696 4349 heartbeater.cc:503] Master 127.21.58.126:37789 was elected leader, sending a full tablet report...
W20240502 14:16:58.221289 4615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:58.221649 4615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:58.221776 4615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:58.229276 4615 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:58.229383 4615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:58.229523 4615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:16:58.238233 4615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:58.239626 4615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:58.241477 4615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:58.252535 4621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:58.253062 4622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:58.253832 4624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:58.254119 4615 server_base.cc:1034] running on GCE node
I20240502 14:16:58.254621 4615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:58.255476 4615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:58.256687 4615 hybrid_clock.cc:648] HybridClock initialized: now 1714659418256642 us; error 42 us; skew 500 ppm
I20240502 14:16:58.256975 4615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:58.259034 4615 webserver.cc:460] Webserver started at http://127.21.58.69:33121/ using document root <none> and password file <none>
I20240502 14:16:58.259727 4615 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:58.259850 4615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:58.260128 4615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:58.262542 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cddf58ccc0c64875a1fb935bf72412ea"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:16:58.263299 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cddf58ccc0c64875a1fb935bf72412ea"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:16:58.268159 4615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 14:16:58.271292 4630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:58.272521 4615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:16:58.272682 4615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cddf58ccc0c64875a1fb935bf72412ea"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:16:58.272884 4615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:58.285950 4615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:58.286641 4615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:58.286866 4615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:58.287761 4615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:58.289076 4615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:58.289158 4615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:58.289234 4615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:58.289286 4615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:58.318828 4615 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:37277
I20240502 14:16:58.318869 4742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:37277 every 8 connection(s)
I20240502 14:16:58.320777 4615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:16:58.323348 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 4615
I20240502 14:16:58.323536 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:16:58.327870 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.70:0
--local_ip_for_outbound_sockets=127.21.58.70
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:16:58.336390 4743 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:16:58.336706 4743 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:58.337705 4743 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:16:58.386358 4030 ts_manager.cc:194] Registered new tserver with Master: cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69:37277)
I20240502 14:16:58.387343 4030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:56237
I20240502 14:16:58.518131 4481 heartbeater.cc:503] Master 127.21.58.126:37789 was elected leader, sending a full tablet report...
W20240502 14:16:58.657284 4746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:16:58.657547 4746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:16:58.657701 4746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:16:58.664739 4746 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:16:58.664876 4746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:16:58.665020 4746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.70
I20240502 14:16:58.673527 4746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.58.70
--webserver_port=0
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.70
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:16:58.674697 4746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:16:58.676381 4746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:16:58.687768 4752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:58.688225 4755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:16:58.688848 4753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:16:58.689169 4746 server_base.cc:1034] running on GCE node
I20240502 14:16:58.689672 4746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:16:58.690460 4746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:16:58.691668 4746 hybrid_clock.cc:648] HybridClock initialized: now 1714659418691594 us; error 49 us; skew 500 ppm
I20240502 14:16:58.691964 4746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:16:58.697605 4746 webserver.cc:460] Webserver started at http://127.21.58.70:36437/ using document root <none> and password file <none>
I20240502 14:16:58.698272 4746 fs_manager.cc:359] Metadata directory not provided
I20240502 14:16:58.698386 4746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:16:58.698680 4746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:16:58.701074 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "146ef337cd9542698f96f62bed9a78ba"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:16:58.701819 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "146ef337cd9542698f96f62bed9a78ba"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:16:58.706406 4746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240502 14:16:58.709424 4761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:58.710532 4746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:58.710671 4746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "146ef337cd9542698f96f62bed9a78ba"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:16:58.710861 4746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:16:58.728111 4746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:16:58.728825 4746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:16:58.729063 4746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:16:58.729851 4746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:16:58.731158 4746 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:16:58.731240 4746 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:58.731317 4746 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:16:58.731369 4746 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:16:58.761077 4746 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.70:44877
I20240502 14:16:58.761123 4873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.70:44877 every 8 connection(s)
I20240502 14:16:58.762497 4746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:16:58.771929 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 4746
I20240502 14:16:58.772126 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:16:58.774075 4874 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:16:58.774346 4874 heartbeater.cc:464] Registering TS with master...
I20240502 14:16:58.775099 4874 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:16:58.818545 4031 ts_manager.cc:194] Registered new tserver with Master: 146ef337cd9542698f96f62bed9a78ba (127.21.58.70:44877)
I20240502 14:16:58.819525 4031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.70:51763
I20240502 14:16:58.824931 21737 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:16:58.840251 21737 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:16:58.840417 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 4746
I20240502 14:16:58.848223 4869 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:58.928905 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 4746
I20240502 14:16:58.944622 4612 heartbeater.cc:503] Master 127.21.58.126:37789 was elected leader, sending a full tablet report...
I20240502 14:16:58.945986 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 4484
I20240502 14:16:58.953362 4607 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:59.038394 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 4484
I20240502 14:16:59.055236 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 4615
I20240502 14:16:59.062922 4738 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:16:59.141069 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 4615
I20240502 14:17:02.910952 4031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59914:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 14:17:02.917292 4031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:17:02.958104 4152 tablet_service.cc:1467] Processing CreateTablet for tablet 6d51ca94fecc4296b23924332c7df8bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.958851 4150 tablet_service.cc:1467] Processing CreateTablet for tablet efe680d0451a49f4b20da6ae2032223a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.959715 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe680d0451a49f4b20da6ae2032223a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.959568 4284 tablet_service.cc:1467] Processing CreateTablet for tablet 6d51ca94fecc4296b23924332c7df8bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.958186 4151 tablet_service.cc:1467] Processing CreateTablet for tablet 326ac01b1c6c4107bb9b682ddb5af56f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.960600 4283 tablet_service.cc:1467] Processing CreateTablet for tablet 326ac01b1c6c4107bb9b682ddb5af56f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.960945 4284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d51ca94fecc4296b23924332c7df8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.962399 4283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326ac01b1c6c4107bb9b682ddb5af56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.963204 4151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326ac01b1c6c4107bb9b682ddb5af56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.959740 4282 tablet_service.cc:1467] Processing CreateTablet for tablet efe680d0451a49f4b20da6ae2032223a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.962786 4415 tablet_service.cc:1467] Processing CreateTablet for tablet 6d51ca94fecc4296b23924332c7df8bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.963768 4413 tablet_service.cc:1467] Processing CreateTablet for tablet efe680d0451a49f4b20da6ae2032223a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.963968 4282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe680d0451a49f4b20da6ae2032223a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.963857 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 326ac01b1c6c4107bb9b682ddb5af56f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:02.964529 4413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe680d0451a49f4b20da6ae2032223a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.964550 4152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d51ca94fecc4296b23924332c7df8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.964965 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326ac01b1c6c4107bb9b682ddb5af56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.967752 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d51ca94fecc4296b23924332c7df8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:02.983492 4900 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:02.986182 4902 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:02.987365 4900 tablet_bootstrap.cc:654] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:02.988752 4900 log.cc:824] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:02.989528 4902 tablet_bootstrap.cc:654] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:02.990715 4902 log.cc:824] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:02.991864 4900 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:02.992216 4900 ts_tablet_manager.cc:1367] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:17:02.996222 4901 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:02.999827 4902 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:02.999847 4901 tablet_bootstrap.cc:654] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:03.000136 4902 ts_tablet_manager.cc:1367] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240502 14:17:03.000878 4901 log.cc:824] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:03.006079 4901 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:03.006000 4900 raft_consensus.cc:348] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.006311 4901 ts_tablet_manager.cc:1367] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:17:03.006425 4900 raft_consensus.cc:374] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.006528 4900 raft_consensus.cc:729] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.007254 4900 consensus_queue.cc:260] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.007256 4902 raft_consensus.cc:348] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.007683 4902 raft_consensus.cc:374] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.007795 4902 raft_consensus.cc:729] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.009181 4900 ts_tablet_manager.cc:1398] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.017s user 0.005s sys 0.007s
I20240502 14:17:03.009604 4900 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:03.009346 4902 consensus_queue.cc:260] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.011106 4900 tablet_bootstrap.cc:654] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:03.013983 4902 ts_tablet_manager.cc:1398] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20240502 14:17:03.014375 4902 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:03.015904 4902 tablet_bootstrap.cc:654] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:03.016216 4901 raft_consensus.cc:348] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.016674 4901 raft_consensus.cc:374] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.016798 4901 raft_consensus.cc:729] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.017634 4901 consensus_queue.cc:260] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.019232 4900 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:03.019423 4900 ts_tablet_manager.cc:1367] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:17:03.020238 4902 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:03.020359 4902 ts_tablet_manager.cc:1367] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:17:03.020339 4900 raft_consensus.cc:348] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.020556 4900 raft_consensus.cc:374] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.020658 4900 raft_consensus.cc:729] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.020870 4900 consensus_queue.cc:260] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.021214 4902 raft_consensus.cc:348] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.021417 4902 raft_consensus.cc:374] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.021476 4900 ts_tablet_manager.cc:1398] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:03.021504 4902 raft_consensus.cc:729] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.021766 4900 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:03.021718 4902 consensus_queue.cc:260] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.022128 4901 ts_tablet_manager.cc:1398] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20240502 14:17:03.022521 4901 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:03.023038 4902 ts_tablet_manager.cc:1398] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:17:03.023353 4902 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:03.023363 4900 tablet_bootstrap.cc:654] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:03.024087 4901 tablet_bootstrap.cc:654] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:03.024876 4902 tablet_bootstrap.cc:654] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
W20240502 14:17:03.027428 4218 tablet.cc:2369] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:17:03.030673 4902 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:03.030853 4902 ts_tablet_manager.cc:1367] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:17:03.031833 4900 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:03.031755 4902 raft_consensus.cc:348] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.031991 4900 ts_tablet_manager.cc:1367] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 14:17:03.032040 4902 raft_consensus.cc:374] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.032135 4902 raft_consensus.cc:729] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.032419 4902 consensus_queue.cc:260] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.032668 4900 raft_consensus.cc:348] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.033036 4900 raft_consensus.cc:374] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.033111 4900 raft_consensus.cc:729] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.033164 4902 ts_tablet_manager.cc:1398] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:17:03.033421 4900 consensus_queue.cc:260] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.033926 4900 ts_tablet_manager.cc:1398] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:03.035351 4901 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:03.035488 4901 ts_tablet_manager.cc:1367] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240502 14:17:03.036370 4906 raft_consensus.cc:482] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:03.036545 4906 raft_consensus.cc:504] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.038030 4906 leader_election.cc:288] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:03.038517 4901 raft_consensus.cc:348] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.039189 4901 raft_consensus.cc:374] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.039314 4901 raft_consensus.cc:729] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.039556 4901 consensus_queue.cc:260] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.040468 4901 ts_tablet_manager.cc:1398] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:17:03.040705 4901 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:03.042363 4901 tablet_bootstrap.cc:654] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:03.045010 4901 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:03.045154 4901 ts_tablet_manager.cc:1367] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:17:03.045742 4901 raft_consensus.cc:348] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.045991 4901 raft_consensus.cc:374] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:03.046070 4901 raft_consensus.cc:729] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:03.046391 4901 consensus_queue.cc:260] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.047129 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe680d0451a49f4b20da6ae2032223a" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551" is_pre_election: true
I20240502 14:17:03.047367 4901 ts_tablet_manager.cc:1398] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:03.047644 4435 raft_consensus.cc:2454] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 0.
I20240502 14:17:03.048166 4237 leader_election.cc:302] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:03.048506 4906 raft_consensus.cc:2781] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:03.048636 4906 raft_consensus.cc:482] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:03.048517 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe680d0451a49f4b20da6ae2032223a" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff" is_pre_election: true
I20240502 14:17:03.048732 4906 raft_consensus.cc:3037] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.049005 4172 raft_consensus.cc:2454] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 0.
I20240502 14:17:03.050935 4906 raft_consensus.cc:504] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.051504 4906 leader_election.cc:288] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:03.052047 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe680d0451a49f4b20da6ae2032223a" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:03.052232 4435 raft_consensus.cc:3037] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.054169 4435 raft_consensus.cc:2454] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 1.
I20240502 14:17:03.054486 4237 leader_election.cc:302] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:03.054769 4906 raft_consensus.cc:2781] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:03.055428 4906 raft_consensus.cc:686] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Running, Role: LEADER
I20240502 14:17:03.056033 4906 consensus_queue.cc:237] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.060794 4907 raft_consensus.cc:482] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:03.061007 4907 raft_consensus.cc:504] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.062994 4907 leader_election.cc:288] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:03.064244 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe680d0451a49f4b20da6ae2032223a" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:03.064440 4172 raft_consensus.cc:3037] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.065488 4029 catalog_manager.cc:5484] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:03.068979 4172 raft_consensus.cc:2454] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 1.
I20240502 14:17:03.071805 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326ac01b1c6c4107bb9b682ddb5af56f" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff" is_pre_election: true
I20240502 14:17:03.071985 4172 raft_consensus.cc:2454] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:03.071862 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326ac01b1c6c4107bb9b682ddb5af56f" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5" is_pre_election: true
I20240502 14:17:03.072194 4304 raft_consensus.cc:2454] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:03.072373 4371 leader_election.cc:302] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:03.072687 4907 raft_consensus.cc:2781] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:03.072790 4907 raft_consensus.cc:482] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:03.072882 4907 raft_consensus.cc:3037] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.074651 4907 raft_consensus.cc:504] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.075179 4907 leader_election.cc:288] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Requested vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:03.075492 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326ac01b1c6c4107bb9b682ddb5af56f" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:03.075516 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326ac01b1c6c4107bb9b682ddb5af56f" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:03.075629 4304 raft_consensus.cc:3037] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.075666 4172 raft_consensus.cc:3037] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.077278 4172 raft_consensus.cc:2454] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 1.
I20240502 14:17:03.077416 4304 raft_consensus.cc:2454] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 1.
I20240502 14:17:03.077582 4371 leader_election.cc:302] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:03.077826 4907 raft_consensus.cc:2781] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:03.078228 4907 raft_consensus.cc:686] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Becoming Leader. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Running, Role: LEADER
I20240502 14:17:03.078641 4907 consensus_queue.cc:237] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.082571 4029 catalog_manager.cc:5484] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } }
I20240502 14:17:03.094744 4907 raft_consensus.cc:482] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:03.094913 4907 raft_consensus.cc:504] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.095386 4907 leader_election.cc:288] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:03.095782 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d51ca94fecc4296b23924332c7df8bc" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff" is_pre_election: true
I20240502 14:17:03.095842 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d51ca94fecc4296b23924332c7df8bc" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5" is_pre_election: true
I20240502 14:17:03.096004 4172 raft_consensus.cc:2454] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:03.096035 4304 raft_consensus.cc:2454] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:03.096340 4371 leader_election.cc:302] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:03.096566 4907 raft_consensus.cc:2781] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:03.096654 4907 raft_consensus.cc:482] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:03.096753 4907 raft_consensus.cc:3037] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.098729 4907 raft_consensus.cc:504] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.099262 4907 leader_election.cc:288] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:03.099525 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d51ca94fecc4296b23924332c7df8bc" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:03.099597 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d51ca94fecc4296b23924332c7df8bc" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:03.099660 4172 raft_consensus.cc:3037] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.099797 4304 raft_consensus.cc:3037] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:03.101395 4172 raft_consensus.cc:2454] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 1.
I20240502 14:17:03.101701 4371 leader_election.cc:302] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:03.101775 4304 raft_consensus.cc:2454] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 1.
I20240502 14:17:03.101923 4907 raft_consensus.cc:2781] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:03.102094 4907 raft_consensus.cc:686] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Becoming Leader. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Running, Role: LEADER
I20240502 14:17:03.102340 4907 consensus_queue.cc:237] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:03.104533 4029 catalog_manager.cc:5484] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } }
W20240502 14:17:03.201525 4482 tablet.cc:2369] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:17:03.226123 4350 tablet.cc:2369] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:17:03.465073 4924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:03.465472 4924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:03.472888 4924 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:17:03.476482 4906 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:03.485677 4906 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 14:17:03.526021 4907 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:03.532089 4921 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:03.578516 4921 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:03.584491 4921 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 220.286 ms
time per row: 0.110143 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:17:04.741806 4029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59914:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 14:17:04.742452 4029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:17:04.749981 4152 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebb0e689b4541f9867ae823046f7844 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.750162 4284 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebb0e689b4541f9867ae823046f7844 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.750396 4152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0e689b4541f9867ae823046f7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.750499 4284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0e689b4541f9867ae823046f7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.750406 4415 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebb0e689b4541f9867ae823046f7844 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.750803 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0e689b4541f9867ae823046f7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.751646 4413 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec22af56f7144fdbe5fc8f361af3555 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.751991 4413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec22af56f7144fdbe5fc8f361af3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.752245 4150 tablet_service.cc:1467] Processing CreateTablet for tablet e295810c85204770b925390874f46d04 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.752611 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e295810c85204770b925390874f46d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.754456 4414 tablet_service.cc:1467] Processing CreateTablet for tablet e295810c85204770b925390874f46d04 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.754814 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e295810c85204770b925390874f46d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.755437 4151 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec22af56f7144fdbe5fc8f361af3555 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.755824 4151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec22af56f7144fdbe5fc8f361af3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.760087 4282 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec22af56f7144fdbe5fc8f361af3555 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.760473 4282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec22af56f7144fdbe5fc8f361af3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.760432 4283 tablet_service.cc:1467] Processing CreateTablet for tablet e295810c85204770b925390874f46d04 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:04.760924 4283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e295810c85204770b925390874f46d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:04.764155 4988 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:04.765763 4988 tablet_bootstrap.cc:654] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.775013 4990 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:04.776288 4989 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:04.776347 4988 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:04.776489 4988 ts_tablet_manager.cc:1367] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.006s
I20240502 14:17:04.776691 4990 tablet_bootstrap.cc:654] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.777153 4988 raft_consensus.cc:348] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.777310 4988 raft_consensus.cc:374] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.777398 4988 raft_consensus.cc:729] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.777662 4988 consensus_queue.cc:260] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.778087 4989 tablet_bootstrap.cc:654] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.778297 4988 ts_tablet_manager.cc:1398] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:17:04.778527 4988 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:04.780032 4988 tablet_bootstrap.cc:654] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.780198 4989 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:04.780324 4989 ts_tablet_manager.cc:1367] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:17:04.781105 4989 raft_consensus.cc:348] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.781308 4989 raft_consensus.cc:374] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.781416 4989 raft_consensus.cc:729] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.781759 4988 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:04.781656 4989 consensus_queue.cc:260] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.781912 4988 ts_tablet_manager.cc:1367] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:17:04.782505 4988 raft_consensus.cc:348] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.782651 4989 ts_tablet_manager.cc:1398] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:04.782724 4988 raft_consensus.cc:374] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.782802 4988 raft_consensus.cc:729] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.782925 4989 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:04.783033 4988 consensus_queue.cc:260] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.783587 4988 ts_tablet_manager.cc:1398] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:04.783861 4988 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:04.784734 4989 tablet_bootstrap.cc:654] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.785149 4988 tablet_bootstrap.cc:654] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.786285 4990 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:04.786422 4990 ts_tablet_manager.cc:1367] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 14:17:04.786428 4989 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:04.786589 4989 ts_tablet_manager.cc:1367] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:17:04.787113 4990 raft_consensus.cc:348] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.787364 4990 raft_consensus.cc:374] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.787338 4989 raft_consensus.cc:348] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.787508 4990 raft_consensus.cc:729] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.787582 4989 raft_consensus.cc:374] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.787690 4989 raft_consensus.cc:729] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.787914 4989 consensus_queue.cc:260] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.787789 4990 consensus_queue.cc:260] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.788350 4989 ts_tablet_manager.cc:1398] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:04.788671 4990 ts_tablet_manager.cc:1398] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:17:04.789090 4990 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:04.789170 4989 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:04.790578 4989 tablet_bootstrap.cc:654] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.791018 4990 tablet_bootstrap.cc:654] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.792577 4988 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:04.792699 4988 ts_tablet_manager.cc:1367] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 14:17:04.793277 4988 raft_consensus.cc:348] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.793459 4988 raft_consensus.cc:374] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.793561 4988 raft_consensus.cc:729] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.793763 4988 consensus_queue.cc:260] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.794134 4988 ts_tablet_manager.cc:1398] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:17:04.794713 4990 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:04.794849 4990 ts_tablet_manager.cc:1367] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:17:04.795571 4990 raft_consensus.cc:348] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.795766 4990 raft_consensus.cc:374] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.795838 4990 raft_consensus.cc:729] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.796077 4990 consensus_queue.cc:260] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.796540 4990 ts_tablet_manager.cc:1398] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:17:04.796810 4990 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:04.797895 4989 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:04.798023 4989 ts_tablet_manager.cc:1367] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 14:17:04.798105 4990 tablet_bootstrap.cc:654] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:04.798710 4989 raft_consensus.cc:348] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.798852 4989 raft_consensus.cc:374] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.798949 4989 raft_consensus.cc:729] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.799230 4989 consensus_queue.cc:260] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.799796 4989 ts_tablet_manager.cc:1398] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:17:04.800120 4990 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:04.800261 4990 ts_tablet_manager.cc:1367] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:17:04.800814 4990 raft_consensus.cc:348] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.800966 4990 raft_consensus.cc:374] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:04.801059 4990 raft_consensus.cc:729] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:04.801291 4990 consensus_queue.cc:260] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.801760 4990 ts_tablet_manager.cc:1398] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:17:04.891847 4906 raft_consensus.cc:482] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:04.892056 4906 raft_consensus.cc:504] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.892653 4906 leader_election.cc:288] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:04.893152 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec22af56f7144fdbe5fc8f361af3555" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551" is_pre_election: true
I20240502 14:17:04.893173 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec22af56f7144fdbe5fc8f361af3555" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff" is_pre_election: true
I20240502 14:17:04.893358 4171 raft_consensus.cc:2454] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 0.
I20240502 14:17:04.893361 4435 raft_consensus.cc:2454] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 0.
I20240502 14:17:04.893788 4237 leader_election.cc:302] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:04.894100 4906 raft_consensus.cc:2781] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:04.894173 4906 raft_consensus.cc:482] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:04.894260 4906 raft_consensus.cc:3037] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.896092 4906 raft_consensus.cc:504] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.896688 4906 leader_election.cc:288] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:04.896982 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec22af56f7144fdbe5fc8f361af3555" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:04.897058 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec22af56f7144fdbe5fc8f361af3555" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:04.897132 4171 raft_consensus.cc:3037] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.897240 4435 raft_consensus.cc:3037] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.898931 4435 raft_consensus.cc:2454] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 1.
I20240502 14:17:04.899039 4171 raft_consensus.cc:2454] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 1.
I20240502 14:17:04.899376 4237 leader_election.cc:302] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:04.899608 4997 raft_consensus.cc:482] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:04.899677 4906 raft_consensus.cc:2781] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:04.899761 4997 raft_consensus.cc:504] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.900089 4906 raft_consensus.cc:686] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Running, Role: LEADER
I20240502 14:17:04.900353 4906 consensus_queue.cc:237] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.901207 4997 leader_election.cc:288] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:04.915567 4029 catalog_manager.cc:5484] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:04.922047 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551" is_pre_election: true
I20240502 14:17:04.922250 4978 raft_consensus.cc:482] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:04.922408 4978 raft_consensus.cc:504] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.922741 4435 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 0.
I20240502 14:17:04.923095 4978 leader_election.cc:288] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:04.923542 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5" is_pre_election: true
I20240502 14:17:04.923540 4105 leader_election.cc:302] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:04.923815 4304 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:04.924080 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff" is_pre_election: true
I20240502 14:17:04.924144 4997 raft_consensus.cc:2781] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:04.924270 4171 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:04.924475 4997 raft_consensus.cc:482] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:04.924580 4997 raft_consensus.cc:3037] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.925289 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5" is_pre_election: true
I20240502 14:17:04.925475 4304 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 0.
I20240502 14:17:04.926604 4997 raft_consensus.cc:504] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.927189 4997 leader_election.cc:288] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 election: Requested vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:04.927488 4371 leader_election.cc:302] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:04.927613 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:04.927757 4978 raft_consensus.cc:2781] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:04.927793 4304 raft_consensus.cc:3037] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.927874 4978 raft_consensus.cc:482] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:04.927968 4978 raft_consensus.cc:3037] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.928615 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:04.929813 4304 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 1.
I20240502 14:17:04.930071 4978 raft_consensus.cc:504] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.930145 4108 leader_election.cc:302] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff; no voters:
I20240502 14:17:04.930406 4435 raft_consensus.cc:2379] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 79041fe14ee64478b7601e9b2230ccff in current term 1: Already voted for candidate 8f899bd3e9a148f5923c0277de1b3551 in this term.
I20240502 14:17:04.930420 4997 raft_consensus.cc:2781] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:04.930770 4978 leader_election.cc:288] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:04.931224 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:04.931529 4304 raft_consensus.cc:2379] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f899bd3e9a148f5923c0277de1b3551 in current term 1: Already voted for candidate 79041fe14ee64478b7601e9b2230ccff in this term.
I20240502 14:17:04.931614 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:04.935267 4997 raft_consensus.cc:686] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Becoming Leader. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Running, Role: LEADER
I20240502 14:17:04.935633 4997 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.937827 4371 leader_election.cc:302] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f899bd3e9a148f5923c0277de1b3551; no voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff
I20240502 14:17:04.938081 4978 raft_consensus.cc:2726] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:17:04.938974 4029 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: term changed from 0 to 1, leader changed from <none> to 79041fe14ee64478b7601e9b2230ccff (127.21.58.65). New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:04.982036 4906 raft_consensus.cc:482] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:04.982257 4906 raft_consensus.cc:504] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.982898 4906 leader_election.cc:288] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:04.983484 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e295810c85204770b925390874f46d04" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff" is_pre_election: true
I20240502 14:17:04.983556 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e295810c85204770b925390874f46d04" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551" is_pre_election: true
I20240502 14:17:04.983750 4435 raft_consensus.cc:2454] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 0.
I20240502 14:17:04.983729 4171 raft_consensus.cc:2454] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 0.
I20240502 14:17:04.984192 4237 leader_election.cc:302] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:04.984465 4906 raft_consensus.cc:2781] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:04.984580 4906 raft_consensus.cc:482] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:04.984668 4906 raft_consensus.cc:3037] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.986575 4906 raft_consensus.cc:504] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.987112 4906 leader_election.cc:288] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:04.987586 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e295810c85204770b925390874f46d04" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:04.987766 4171 raft_consensus.cc:3037] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.987707 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e295810c85204770b925390874f46d04" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:04.987890 4435 raft_consensus.cc:3037] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:04.990538 4171 raft_consensus.cc:2454] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 1.
I20240502 14:17:04.990976 4240 leader_election.cc:302] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff; no voters:
I20240502 14:17:04.991412 4435 raft_consensus.cc:2454] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 1.
I20240502 14:17:04.991760 4906 raft_consensus.cc:2781] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:04.991966 4906 raft_consensus.cc:686] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Becoming Leader. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Running, Role: LEADER
I20240502 14:17:04.992477 4906 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:04.995117 4029 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:05.289269 4906 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:05.296864 4906 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:17:05.384634 5010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:05.384935 5010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:05.392104 5010 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:17:05.449622 4906 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:05.473011 4997 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:05.475867 5040 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:05.536512 5044 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
INSERT report
rows total: 2000
time total: 256.659 ms
time per row: 0.128329 ms
I20240502 14:17:05.787734 4028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59914:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 14:17:05.788489 4028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:17:05.796105 4151 tablet_service.cc:1467] Processing CreateTablet for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.796269 4284 tablet_service.cc:1467] Processing CreateTablet for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.796579 4151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.796694 4284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.797866 4150 tablet_service.cc:1467] Processing CreateTablet for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.798007 4282 tablet_service.cc:1467] Processing CreateTablet for tablet 4149880ddb9d4e319c6386302d89fe0a (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.798252 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a9469b9baa42f9868fbff0b4d9f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.798413 4282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4149880ddb9d4e319c6386302d89fe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.799804 4152 tablet_service.cc:1467] Processing CreateTablet for tablet 4149880ddb9d4e319c6386302d89fe0a (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.799801 4415 tablet_service.cc:1467] Processing CreateTablet for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.800184 4152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4149880ddb9d4e319c6386302d89fe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.800199 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.803421 4283 tablet_service.cc:1467] Processing CreateTablet for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.803725 4283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a9469b9baa42f9868fbff0b4d9f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.807720 4413 tablet_service.cc:1467] Processing CreateTablet for tablet 4149880ddb9d4e319c6386302d89fe0a (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.808157 4413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4149880ddb9d4e319c6386302d89fe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.808483 5071 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:05.810173 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:05.810315 5071 tablet_bootstrap.cc:654] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.810597 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a9469b9baa42f9868fbff0b4d9f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:05.821597 5071 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:05.821758 5071 ts_tablet_manager.cc:1367] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240502 14:17:05.821990 5072 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:05.822639 5071 raft_consensus.cc:348] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.822863 5071 raft_consensus.cc:374] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.822970 5071 raft_consensus.cc:729] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.823061 5074 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:05.823275 5071 consensus_queue.cc:260] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.823936 5071 ts_tablet_manager.cc:1398] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:05.824069 5072 tablet_bootstrap.cc:654] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.824195 5071 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:05.825688 5074 tablet_bootstrap.cc:654] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.825932 5071 tablet_bootstrap.cc:654] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.831027 5074 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:05.831187 5074 ts_tablet_manager.cc:1367] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240502 14:17:05.831480 5072 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:05.831619 5072 ts_tablet_manager.cc:1367] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.004s
I20240502 14:17:05.831673 5071 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:05.831837 5071 ts_tablet_manager.cc:1367] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240502 14:17:05.831971 5074 raft_consensus.cc:348] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.832186 5074 raft_consensus.cc:374] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.832278 5074 raft_consensus.cc:729] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.832300 5072 raft_consensus.cc:348] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.832489 5072 raft_consensus.cc:374] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.832584 5072 raft_consensus.cc:729] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.832506 5074 consensus_queue.cc:260] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.832551 5071 raft_consensus.cc:348] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.832749 5071 raft_consensus.cc:374] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.832829 5071 raft_consensus.cc:729] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.832823 5072 consensus_queue.cc:260] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.833055 5071 consensus_queue.cc:260] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.833401 5072 ts_tablet_manager.cc:1398] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:05.833662 5071 ts_tablet_manager.cc:1398] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:17:05.833930 5071 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:05.834398 5074 ts_tablet_manager.cc:1398] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:17:05.834656 5074 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:05.835377 5071 tablet_bootstrap.cc:654] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.837235 5074 tablet_bootstrap.cc:654] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.833642 5072 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:05.840230 5072 tablet_bootstrap.cc:654] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.841814 5072 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:05.841964 5072 ts_tablet_manager.cc:1367] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:17:05.842478 5072 raft_consensus.cc:348] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.842635 5072 raft_consensus.cc:374] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.842707 5072 raft_consensus.cc:729] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.842941 5072 consensus_queue.cc:260] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.843529 5072 ts_tablet_manager.cc:1398] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:17:05.843815 5072 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:05.845041 5072 tablet_bootstrap.cc:654] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.846691 5074 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:05.846777 5071 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff: No bootstrap required, opened a new log
I20240502 14:17:05.846850 5074 ts_tablet_manager.cc:1367] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240502 14:17:05.846938 5071 ts_tablet_manager.cc:1367] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240502 14:17:05.847651 5072 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: No bootstrap required, opened a new log
I20240502 14:17:05.847779 5072 ts_tablet_manager.cc:1367] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:17:05.847635 5074 raft_consensus.cc:348] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.847875 5074 raft_consensus.cc:374] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.847963 5074 raft_consensus.cc:729] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.847959 5071 raft_consensus.cc:348] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.848146 5071 raft_consensus.cc:374] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.848242 5071 raft_consensus.cc:729] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.848202 5074 consensus_queue.cc:260] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.848487 5071 consensus_queue.cc:260] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.848855 5074 ts_tablet_manager.cc:1398] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:17:05.848987 5071 ts_tablet_manager.cc:1398] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:05.848484 5072 raft_consensus.cc:348] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.849156 5074 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:05.849180 5072 raft_consensus.cc:374] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.849261 5072 raft_consensus.cc:729] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.849483 5072 consensus_queue.cc:260] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.850044 5072 ts_tablet_manager.cc:1398] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:05.850606 5074 tablet_bootstrap.cc:654] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:05.853747 5074 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: No bootstrap required, opened a new log
I20240502 14:17:05.853875 5074 ts_tablet_manager.cc:1367] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:17:05.854449 5074 raft_consensus.cc:348] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.854602 5074 raft_consensus.cc:374] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:05.854692 5074 raft_consensus.cc:729] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: FOLLOWER
I20240502 14:17:05.854888 5074 consensus_queue.cc:260] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.855247 4997 raft_consensus.cc:482] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:05.855319 5074 ts_tablet_manager.cc:1398] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:17:05.855365 4997 raft_consensus.cc:504] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.855903 4997 leader_election.cc:288] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:05.856321 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a9469b9baa42f9868fbff0b4d9f0a4" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5" is_pre_election: true
I20240502 14:17:05.856348 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a9469b9baa42f9868fbff0b4d9f0a4" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551" is_pre_election: true
I20240502 14:17:05.856487 4304 raft_consensus.cc:2454] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 0.
I20240502 14:17:05.856523 4434 raft_consensus.cc:2454] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 0.
I20240502 14:17:05.856768 4108 leader_election.cc:302] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff; no voters:
I20240502 14:17:05.857029 4997 raft_consensus.cc:2781] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:05.857147 4997 raft_consensus.cc:482] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:05.857235 4997 raft_consensus.cc:3037] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:05.859233 4997 raft_consensus.cc:504] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.859738 4997 leader_election.cc:288] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 election: Requested vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:05.860097 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a9469b9baa42f9868fbff0b4d9f0a4" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:05.860149 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a9469b9baa42f9868fbff0b4d9f0a4" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:05.860257 4304 raft_consensus.cc:3037] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:05.860311 4434 raft_consensus.cc:3037] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:05.862111 4304 raft_consensus.cc:2454] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 1.
I20240502 14:17:05.862202 4434 raft_consensus.cc:2454] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 1.
I20240502 14:17:05.862462 4108 leader_election.cc:302] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff; no voters:
I20240502 14:17:05.862711 4997 raft_consensus.cc:2781] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:05.862854 4997 raft_consensus.cc:686] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Becoming Leader. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Running, Role: LEADER
I20240502 14:17:05.863119 4997 consensus_queue.cc:237] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.865466 4029 catalog_manager.cc:5484] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: term changed from 0 to 1, leader changed from <none> to 79041fe14ee64478b7601e9b2230ccff (127.21.58.65). New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:05.920481 4978 raft_consensus.cc:482] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:05.920666 4978 raft_consensus.cc:504] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.921232 4978 leader_election.cc:288] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:05.921730 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd6d3a24c3f47bfa0dbf306d787b4a0" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff" is_pre_election: true
I20240502 14:17:05.921800 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd6d3a24c3f47bfa0dbf306d787b4a0" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5" is_pre_election: true
I20240502 14:17:05.922046 4171 raft_consensus.cc:2454] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:05.922066 4304 raft_consensus.cc:2454] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 0.
I20240502 14:17:05.922467 4371 leader_election.cc:302] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:05.922747 4978 raft_consensus.cc:2781] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:05.922835 4978 raft_consensus.cc:482] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:05.922981 4978 raft_consensus.cc:3037] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:05.924768 4978 raft_consensus.cc:504] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.925308 4978 leader_election.cc:288] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:05.925642 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd6d3a24c3f47bfa0dbf306d787b4a0" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:05.925846 4171 raft_consensus.cc:3037] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:05.925722 4303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd6d3a24c3f47bfa0dbf306d787b4a0" candidate_uuid: "8f899bd3e9a148f5923c0277de1b3551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:05.925987 4303 raft_consensus.cc:3037] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:05.928683 4171 raft_consensus.cc:2454] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 1.
I20240502 14:17:05.928879 4303 raft_consensus.cc:2454] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f899bd3e9a148f5923c0277de1b3551 in term 1.
I20240502 14:17:05.929096 4371 leader_election.cc:302] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:05.929351 4978 raft_consensus.cc:2781] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:05.929500 4978 raft_consensus.cc:686] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Becoming Leader. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Running, Role: LEADER
I20240502 14:17:05.929757 4978 consensus_queue.cc:237] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:05.932471 4029 catalog_manager.cc:5484] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } }
I20240502 14:17:06.044054 4997 raft_consensus.cc:482] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:06.044276 4997 raft_consensus.cc:504] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:06.044903 4997 leader_election.cc:288] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:06.045440 4303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5" is_pre_election: true
I20240502 14:17:06.045475 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551" is_pre_election: true
I20240502 14:17:06.045702 4303 raft_consensus.cc:2454] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 0.
I20240502 14:17:06.045711 4434 raft_consensus.cc:2454] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 0.
I20240502 14:17:06.046093 4108 leader_election.cc:302] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff; no voters:
I20240502 14:17:06.046397 4997 raft_consensus.cc:2781] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:06.046489 4997 raft_consensus.cc:482] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:06.046559 4997 raft_consensus.cc:3037] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:06.048719 4997 raft_consensus.cc:504] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:06.049211 4997 leader_election.cc:288] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 election: Requested vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:06.049585 4303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:06.049676 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a" candidate_uuid: "79041fe14ee64478b7601e9b2230ccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:06.049762 4303 raft_consensus.cc:3037] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:06.049875 4434 raft_consensus.cc:3037] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:06.051687 4303 raft_consensus.cc:2454] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 1.
I20240502 14:17:06.051791 4434 raft_consensus.cc:2454] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79041fe14ee64478b7601e9b2230ccff in term 1.
I20240502 14:17:06.052047 4108 leader_election.cc:302] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff; no voters:
I20240502 14:17:06.052351 4997 raft_consensus.cc:2781] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:06.052533 4997 raft_consensus.cc:686] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Becoming Leader. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Running, Role: LEADER
I20240502 14:17:06.052842 4997 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } }
I20240502 14:17:06.055466 4029 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff reported cstate change: term changed from 0 to 1, leader changed from <none> to 79041fe14ee64478b7601e9b2230ccff (127.21.58.65). New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:06.390283 4978 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:06.396306 5090 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:06.398115 4997 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:06.404953 5045 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:06.448709 5045 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:17:06.450798 5087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:06.451202 5087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:17:06.453338 4997 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:17:06.461205 5087 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 215.817 ms
time per row: 0.107909 ms
I20240502 14:17:06.813619 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.70:44877
--local_ip_for_outbound_sockets=127.21.58.70
--tserver_master_addrs=127.21.58.126:37789
--webserver_port=36437
--webserver_interface=127.21.58.70
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:17:07.149962 5149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:07.150223 5149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:07.150367 5149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:07.157864 5149 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:07.157997 5149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:07.158149 5149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.70
I20240502 14:17:07.167286 5149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.70:44877
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.58.70
--webserver_port=36437
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.70
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:07.168605 5149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:07.170212 5149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:07.182484 5155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:07.182526 5154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:07.183027 5157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:07.183647 5149 server_base.cc:1034] running on GCE node
I20240502 14:17:07.184190 5149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:07.185065 5149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:07.186244 5149 hybrid_clock.cc:648] HybridClock initialized: now 1714659427186187 us; error 49 us; skew 500 ppm
I20240502 14:17:07.186520 5149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:07.188637 5149 webserver.cc:460] Webserver started at http://127.21.58.70:36437/ using document root <none> and password file <none>
I20240502 14:17:07.189317 5149 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:07.189445 5149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:07.194195 5149 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 14:17:07.196821 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:07.198103 5149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 14:17:07.198261 5149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "146ef337cd9542698f96f62bed9a78ba"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:17:07.199223 5149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:07.237169 5149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:07.238014 5149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:07.238278 5149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:07.239202 5149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:07.240677 5149 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:07.240760 5149 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:07.240886 5149 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:07.240939 5149 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:07.272528 5149 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.70:44877
I20240502 14:17:07.272588 5275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.70:44877 every 8 connection(s)
I20240502 14:17:07.273916 5149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:17:07.275981 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 5149
I20240502 14:17:07.276593 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:41023
--local_ip_for_outbound_sockets=127.21.58.68
--tserver_master_addrs=127.21.58.126:37789
--webserver_port=35571
--webserver_interface=127.21.58.68
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:17:07.287593 5276 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:17:07.287874 5276 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:07.288617 5276 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:17:07.290268 4023 ts_manager.cc:194] Re-registered known tserver with Master: 146ef337cd9542698f96f62bed9a78ba (127.21.58.70:44877)
I20240502 14:17:07.291420 4023 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.70:38171
W20240502 14:17:07.611586 5280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:07.611857 5280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:07.611950 5280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:07.619285 5280 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:07.619419 5280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:07.619562 5280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:17:07.629928 5280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:41023
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=35571
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:07.631359 5280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:07.633188 5280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:07.644459 5285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:07.644977 5288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:07.645565 5286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:07.645905 5280 server_base.cc:1034] running on GCE node
I20240502 14:17:07.646440 5280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:07.647332 5280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:07.648527 5280 hybrid_clock.cc:648] HybridClock initialized: now 1714659427648484 us; error 33 us; skew 500 ppm
I20240502 14:17:07.648833 5280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:07.651010 5280 webserver.cc:460] Webserver started at http://127.21.58.68:35571/ using document root <none> and password file <none>
I20240502 14:17:07.651731 5280 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:07.651854 5280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:07.656883 5280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 14:17:07.659461 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:07.660856 5280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:17:07.660987 5280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "42b2fb8410fe4df48fb3d243f813c322"
format_stamp: "Formatted at 2024-05-02 14:16:57 on dist-test-slave-wtgm"
I20240502 14:17:07.661836 5280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:07.688599 5280 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:07.689363 5280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:07.689607 5280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:07.690456 5280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:07.691941 5280 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:07.692008 5280 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:07.692096 5280 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:07.692137 5280 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:07.728104 5280 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:41023
I20240502 14:17:07.728169 5406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:41023 every 8 connection(s)
I20240502 14:17:07.729626 5280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:17:07.739147 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 5280
I20240502 14:17:07.739897 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:37277
--local_ip_for_outbound_sockets=127.21.58.69
--tserver_master_addrs=127.21.58.126:37789
--webserver_port=33121
--webserver_interface=127.21.58.69
--builtin_ntp_servers=127.21.58.84:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:17:07.741902 5407 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:17:07.742228 5407 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:07.743162 5407 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:17:07.744601 4023 ts_manager.cc:194] Re-registered known tserver with Master: 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023)
I20240502 14:17:07.745347 4023 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:34549
W20240502 14:17:08.064546 5410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:08.064802 5410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:08.064925 5410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:08.072137 5410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:08.072288 5410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:08.072409 5410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:17:08.081184 5410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:37277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=33121
--tserver_master_addrs=127.21.58.126:37789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:08.082522 5410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:08.084259 5410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:08.095209 5415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:08.095686 5416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:08.096230 5418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:08.096714 5410 server_base.cc:1034] running on GCE node
I20240502 14:17:08.097208 5410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:08.098033 5410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:08.099220 5410 hybrid_clock.cc:648] HybridClock initialized: now 1714659428099161 us; error 51 us; skew 500 ppm
I20240502 14:17:08.099514 5410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:08.101538 5410 webserver.cc:460] Webserver started at http://127.21.58.69:33121/ using document root <none> and password file <none>
I20240502 14:17:08.102195 5410 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:08.102317 5410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:08.107015 5410 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:17:08.109555 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:08.110713 5410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:08.110868 5410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cddf58ccc0c64875a1fb935bf72412ea"
format_stamp: "Formatted at 2024-05-02 14:16:58 on dist-test-slave-wtgm"
I20240502 14:17:08.111768 5410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:08.132277 5410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:08.133028 5410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:08.133304 5410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:08.134159 5410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:08.135653 5410 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:08.135727 5410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:08.135854 5410 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:08.135908 5410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:08.167919 5410 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:37277
I20240502 14:17:08.167964 5536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:37277 every 8 connection(s)
I20240502 14:17:08.169353 5410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:17:08.169638 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 5410
I20240502 14:17:08.182144 5537 heartbeater.cc:347] Connected to a master server at 127.21.58.126:37789
I20240502 14:17:08.182426 5537 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:08.183209 5537 heartbeater.cc:511] Master 127.21.58.126:37789 requested a full tablet report, sending...
I20240502 14:17:08.184551 4023 ts_manager.cc:194] Re-registered known tserver with Master: cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69:37277)
I20240502 14:17:08.185290 4023 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:52613
I20240502 14:17:08.294124 5276 heartbeater.cc:503] Master 127.21.58.126:37789 was elected leader, sending a full tablet report...
I20240502 14:17:08.641888 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:08.663132 5342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:08.694475 4151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:08.711660 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:08.715173 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:08.725847 5472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:08.748083 5407 heartbeater.cc:503] Master 127.21.58.126:37789 was elected leader, sending a full tablet report...
I20240502 14:17:08.883023 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:08.884250 5342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:08.902036 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:08.917966 4151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:08.920563 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:08.923187 5472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:09.102001 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:09.104292 5342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:09.118213 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:09.124150 5472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:09.126920 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:09.144071 4151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:09.187474 5537 heartbeater.cc:503] Master 127.21.58.126:37789 was elected leader, sending a full tablet report...
I20240502 14:17:09.298729 5342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:09.299386 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:09.306010 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:09.326093 4151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:09.331266 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:09.356109 5472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:09.409440 4434 consensus_queue.cc:237] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:09.412237 4303 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.412787 4171 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.412858 5687 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.413394 5686 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.416357 5686 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.416882 4171 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
W20240502 14:17:09.420400 4368 consensus_peers.cc:485] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 -> Peer 146ef337cd9542698f96f62bed9a78ba (127.21.58.70:44877): Couldn't send request to peer 146ef337cd9542698f96f62bed9a78ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 326ac01b1c6c4107bb9b682ddb5af56f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.421458 4303 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.422457 4027 catalog_manager.cc:5484] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:09.437410 4303 consensus_queue.cc:237] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:09.439879 4171 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.440250 4434 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.440481 5701 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.440939 5088 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.443790 5701 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:09.444670 4434 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:09.450227 4023 catalog_manager.cc:5484] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:09.461769 4171 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
W20240502 14:17:09.478646 4238 consensus_peers.cc:485] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023): Couldn't send request to peer 42b2fb8410fe4df48fb3d243f813c322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ec22af56f7144fdbe5fc8f361af3555. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.478942 4171 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:09.481398 4303 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.481760 4434 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.482442 4997 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.483026 5712 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.486441 4303 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.486795 4434 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.490330 5713 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.490882 4029 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
W20240502 14:17:09.491430 4106 consensus_peers.cc:485] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff -> Peer cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69:37277): Couldn't send request to peer cddf58ccc0c64875a1fb935bf72412ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4149880ddb9d4e319c6386302d89fe0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.505640 4434 consensus_queue.cc:237] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:09.507814 4303 raft_consensus.cc:1261] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.507937 4171 raft_consensus.cc:1261] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:17:09.507984 4368 consensus_peers.cc:485] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 -> Peer 146ef337cd9542698f96f62bed9a78ba (127.21.58.70:44877): Couldn't send request to peer 146ef337cd9542698f96f62bed9a78ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecd6d3a24c3f47bfa0dbf306d787b4a0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.508332 4978 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.508595 5687 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.511145 5686 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.511723 4171 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.511747 4303 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.516672 4023 catalog_manager.cc:5484] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:09.529521 4303 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:09.532253 4171 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:17:09.532647 4238 consensus_peers.cc:485] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023): Couldn't send request to peer 42b2fb8410fe4df48fb3d243f813c322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e295810c85204770b925390874f46d04. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.532722 4434 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.533208 5705 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.533459 5088 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.536146 5088 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:09.539582 4171 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:09.544826 4027 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:09.551031 4434 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:09.565507 4303 consensus_queue.cc:237] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:09.567711 4171 raft_consensus.cc:1261] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.567893 4434 raft_consensus.cc:1261] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.568244 5088 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.568533 5705 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.572523 5705 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.572855 4434 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
W20240502 14:17:09.574393 4238 consensus_peers.cc:485] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 -> Peer cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69:37277): Couldn't send request to peer cddf58ccc0c64875a1fb935bf72412ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efe680d0451a49f4b20da6ae2032223a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.575773 4171 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.577198 4029 catalog_manager.cc:5484] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.589862 4434 consensus_queue.cc:237] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
W20240502 14:17:09.592146 4368 consensus_peers.cc:485] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 -> Peer 146ef337cd9542698f96f62bed9a78ba (127.21.58.70:44877): Couldn't send request to peer 146ef337cd9542698f96f62bed9a78ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d51ca94fecc4296b23924332c7df8bc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.592144 4171 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.592372 4303 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.592653 5686 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.592916 4978 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.595415 5687 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.596094 4171 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.596060 4303 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.600286 4029 catalog_manager.cc:5484] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index -1 to 4, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:09.618944 4171 consensus_queue.cc:237] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:09.620978 4303 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:09.621060 4434 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:17:09.621369 4106 consensus_peers.cc:485] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff -> Peer cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69:37277): Couldn't send request to peer cddf58ccc0c64875a1fb935bf72412ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00a9469b9baa42f9868fbff0b4d9f0a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:09.621593 4997 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.621845 5712 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:09.624507 5712 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.625346 4434 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.627636 4303 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:09.628952 4023 catalog_manager.cc:5484] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index -1 to 4, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:09.977128 5838 ts_tablet_manager.cc:902] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Initiating tablet copy from peer 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:09.989055 5838 tablet_copy_client.cc:308] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:39879
I20240502 14:17:10.004010 5844 ts_tablet_manager.cc:902] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Initiating tablet copy from peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:10.010936 4456 tablet_copy_service.cc:140] P 8f899bd3e9a148f5923c0277de1b3551: Received BeginTabletCopySession request for tablet 326ac01b1c6c4107bb9b682ddb5af56f from peer 146ef337cd9542698f96f62bed9a78ba ({username='slave'} at 127.21.58.70:43701)
I20240502 14:17:10.011229 4456 tablet_copy_service.cc:161] P 8f899bd3e9a148f5923c0277de1b3551: Beginning new tablet copy session on tablet 326ac01b1c6c4107bb9b682ddb5af56f from peer 146ef337cd9542698f96f62bed9a78ba at {username='slave'} at 127.21.58.70:43701: session id = 146ef337cd9542698f96f62bed9a78ba-326ac01b1c6c4107bb9b682ddb5af56f
I20240502 14:17:10.014326 4456 tablet_copy_source_session.cc:205] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.020534 5838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326ac01b1c6c4107bb9b682ddb5af56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.024681 5844 tablet_copy_client.cc:308] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:32867
I20240502 14:17:10.031998 4324 tablet_copy_service.cc:140] P 751046160dd4466eb3f9ffea4128cdc5: Received BeginTabletCopySession request for tablet 5ec22af56f7144fdbe5fc8f361af3555 from peer 42b2fb8410fe4df48fb3d243f813c322 ({username='slave'} at 127.21.58.68:41699)
I20240502 14:17:10.032284 4324 tablet_copy_service.cc:161] P 751046160dd4466eb3f9ffea4128cdc5: Beginning new tablet copy session on tablet 5ec22af56f7144fdbe5fc8f361af3555 from peer 42b2fb8410fe4df48fb3d243f813c322 at {username='slave'} at 127.21.58.68:41699: session id = 42b2fb8410fe4df48fb3d243f813c322-5ec22af56f7144fdbe5fc8f361af3555
I20240502 14:17:10.032750 5838 tablet_copy_client.cc:783] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.033195 5838 tablet_copy_client.cc:647] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.035446 4324 tablet_copy_source_session.cc:205] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.036167 5838 tablet_copy_client.cc:523] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.040174 5838 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Bootstrap starting.
I20240502 14:17:10.040377 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec22af56f7144fdbe5fc8f361af3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.049098 5838 log.cc:824] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:10.050122 5844 tablet_copy_client.cc:783] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.050443 5844 tablet_copy_client.cc:647] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.053053 5844 tablet_copy_client.cc:523] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.056319 5844 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap starting.
I20240502 14:17:10.057009 5851 ts_tablet_manager.cc:902] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: Initiating tablet copy from peer 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:10.058880 5851 tablet_copy_client.cc:308] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:39879
I20240502 14:17:10.059643 4456 tablet_copy_service.cc:140] P 8f899bd3e9a148f5923c0277de1b3551: Received BeginTabletCopySession request for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 from peer 146ef337cd9542698f96f62bed9a78ba ({username='slave'} at 127.21.58.70:43701)
I20240502 14:17:10.059845 4456 tablet_copy_service.cc:161] P 8f899bd3e9a148f5923c0277de1b3551: Beginning new tablet copy session on tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 from peer 146ef337cd9542698f96f62bed9a78ba at {username='slave'} at 127.21.58.70:43701: session id = 146ef337cd9542698f96f62bed9a78ba-ecd6d3a24c3f47bfa0dbf306d787b4a0
I20240502 14:17:10.062767 4456 tablet_copy_source_session.cc:205] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.064054 5851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.067767 5844 log.cc:824] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:10.070550 5851 tablet_copy_client.cc:783] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.070812 5851 tablet_copy_client.cc:647] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.072602 5851 tablet_copy_client.cc:523] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.076079 5851 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: Bootstrap starting.
I20240502 14:17:10.091451 5857 ts_tablet_manager.cc:902] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: Initiating tablet copy from peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:10.099311 5859 ts_tablet_manager.cc:902] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: Initiating tablet copy from peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:10.107220 5838 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.107808 5838 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Bootstrap complete.
I20240502 14:17:10.108402 5838 ts_tablet_manager.cc:1367] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Time spent bootstrapping tablet: real 0.068s user 0.049s sys 0.011s
I20240502 14:17:10.116019 5838 raft_consensus.cc:348] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:10.116710 5838 raft_consensus.cc:729] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 146ef337cd9542698f96f62bed9a78ba, State: Initialized, Role: LEARNER
I20240502 14:17:10.117355 5838 consensus_queue.cc:260] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:10.124552 5864 ts_tablet_manager.cc:902] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: Initiating tablet copy from peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:10.125540 5838 ts_tablet_manager.cc:1398] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20240502 14:17:10.132398 4456 tablet_copy_service.cc:342] P 8f899bd3e9a148f5923c0277de1b3551: Request end of tablet copy session 146ef337cd9542698f96f62bed9a78ba-326ac01b1c6c4107bb9b682ddb5af56f received from {username='slave'} at 127.21.58.70:43701
I20240502 14:17:10.132560 4456 tablet_copy_service.cc:433] P 8f899bd3e9a148f5923c0277de1b3551: ending tablet copy session 146ef337cd9542698f96f62bed9a78ba-326ac01b1c6c4107bb9b682ddb5af56f on tablet 326ac01b1c6c4107bb9b682ddb5af56f with peer 146ef337cd9542698f96f62bed9a78ba
I20240502 14:17:10.135818 5857 tablet_copy_client.cc:308] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:41803
I20240502 14:17:10.138109 5859 tablet_copy_client.cc:308] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:32867
I20240502 14:17:10.139138 4324 tablet_copy_service.cc:140] P 751046160dd4466eb3f9ffea4128cdc5: Received BeginTabletCopySession request for tablet e295810c85204770b925390874f46d04 from peer 42b2fb8410fe4df48fb3d243f813c322 ({username='slave'} at 127.21.58.68:41699)
I20240502 14:17:10.139367 4324 tablet_copy_service.cc:161] P 751046160dd4466eb3f9ffea4128cdc5: Beginning new tablet copy session on tablet e295810c85204770b925390874f46d04 from peer 42b2fb8410fe4df48fb3d243f813c322 at {username='slave'} at 127.21.58.68:41699: session id = 42b2fb8410fe4df48fb3d243f813c322-e295810c85204770b925390874f46d04
I20240502 14:17:10.143553 4324 tablet_copy_source_session.cc:205] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.145334 5859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e295810c85204770b925390874f46d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.152774 5851 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.153234 5851 tablet_bootstrap.cc:492] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: Bootstrap complete.
I20240502 14:17:10.153545 5851 ts_tablet_manager.cc:1367] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: Time spent bootstrapping tablet: real 0.078s user 0.060s sys 0.016s
I20240502 14:17:10.154610 5851 raft_consensus.cc:348] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:10.154958 5851 raft_consensus.cc:729] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 146ef337cd9542698f96f62bed9a78ba, State: Initialized, Role: LEARNER
I20240502 14:17:10.155239 5851 consensus_queue.cc:260] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:10.155864 5851 ts_tablet_manager.cc:1398] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:10.156610 4456 tablet_copy_service.cc:342] P 8f899bd3e9a148f5923c0277de1b3551: Request end of tablet copy session 146ef337cd9542698f96f62bed9a78ba-ecd6d3a24c3f47bfa0dbf306d787b4a0 received from {username='slave'} at 127.21.58.70:43701
I20240502 14:17:10.156747 4456 tablet_copy_service.cc:433] P 8f899bd3e9a148f5923c0277de1b3551: ending tablet copy session 146ef337cd9542698f96f62bed9a78ba-ecd6d3a24c3f47bfa0dbf306d787b4a0 on tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 with peer 146ef337cd9542698f96f62bed9a78ba
I20240502 14:17:10.158967 5859 tablet_copy_client.cc:783] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.160342 4192 tablet_copy_service.cc:140] P 79041fe14ee64478b7601e9b2230ccff: Received BeginTabletCopySession request for tablet 4149880ddb9d4e319c6386302d89fe0a from peer cddf58ccc0c64875a1fb935bf72412ea ({username='slave'} at 127.21.58.69:50797)
I20240502 14:17:10.160593 4192 tablet_copy_service.cc:161] P 79041fe14ee64478b7601e9b2230ccff: Beginning new tablet copy session on tablet 4149880ddb9d4e319c6386302d89fe0a from peer cddf58ccc0c64875a1fb935bf72412ea at {username='slave'} at 127.21.58.69:50797: session id = cddf58ccc0c64875a1fb935bf72412ea-4149880ddb9d4e319c6386302d89fe0a
I20240502 14:17:10.163367 5859 tablet_copy_client.cc:647] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.164321 4192 tablet_copy_source_session.cc:205] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.165155 5859 tablet_copy_client.cc:523] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.165918 5844 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.166570 5844 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap complete.
I20240502 14:17:10.167105 5844 ts_tablet_manager.cc:1367] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Time spent bootstrapping tablet: real 0.111s user 0.061s sys 0.007s
I20240502 14:17:10.168762 5859 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap starting.
I20240502 14:17:10.170015 5857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4149880ddb9d4e319c6386302d89fe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.173703 5844 raft_consensus.cc:348] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:10.174187 5844 raft_consensus.cc:729] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42b2fb8410fe4df48fb3d243f813c322, State: Initialized, Role: LEARNER
I20240502 14:17:10.174736 5844 consensus_queue.cc:260] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:10.178531 5851 ts_tablet_manager.cc:902] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: Initiating tablet copy from peer 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:10.196775 5864 tablet_copy_client.cc:308] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:32867
I20240502 14:17:10.199126 5844 ts_tablet_manager.cc:1398] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Time spent starting tablet: real 0.032s user 0.012s sys 0.008s
I20240502 14:17:10.199975 4324 tablet_copy_service.cc:342] P 751046160dd4466eb3f9ffea4128cdc5: Request end of tablet copy session 42b2fb8410fe4df48fb3d243f813c322-5ec22af56f7144fdbe5fc8f361af3555 received from {username='slave'} at 127.21.58.68:41699
I20240502 14:17:10.200135 4324 tablet_copy_service.cc:433] P 751046160dd4466eb3f9ffea4128cdc5: ending tablet copy session 42b2fb8410fe4df48fb3d243f813c322-5ec22af56f7144fdbe5fc8f361af3555 on tablet 5ec22af56f7144fdbe5fc8f361af3555 with peer 42b2fb8410fe4df48fb3d243f813c322
I20240502 14:17:10.204763 5857 tablet_copy_client.cc:783] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.205119 5857 tablet_copy_client.cc:647] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.206039 4324 tablet_copy_service.cc:140] P 751046160dd4466eb3f9ffea4128cdc5: Received BeginTabletCopySession request for tablet efe680d0451a49f4b20da6ae2032223a from peer cddf58ccc0c64875a1fb935bf72412ea ({username='slave'} at 127.21.58.69:33207)
I20240502 14:17:10.206197 4324 tablet_copy_service.cc:161] P 751046160dd4466eb3f9ffea4128cdc5: Beginning new tablet copy session on tablet efe680d0451a49f4b20da6ae2032223a from peer cddf58ccc0c64875a1fb935bf72412ea at {username='slave'} at 127.21.58.69:33207: session id = cddf58ccc0c64875a1fb935bf72412ea-efe680d0451a49f4b20da6ae2032223a
I20240502 14:17:10.207875 5857 tablet_copy_client.cc:523] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.209919 4324 tablet_copy_source_session.cc:205] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.211325 5864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe680d0451a49f4b20da6ae2032223a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.219640 5864 tablet_copy_client.cc:783] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.219960 5864 tablet_copy_client.cc:647] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.222028 5864 tablet_copy_client.cc:523] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.225154 5857 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap starting.
I20240502 14:17:10.230008 5864 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap starting.
I20240502 14:17:10.235601 5851 tablet_copy_client.cc:308] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:39879
I20240502 14:17:10.236510 4456 tablet_copy_service.cc:140] P 8f899bd3e9a148f5923c0277de1b3551: Received BeginTabletCopySession request for tablet 6d51ca94fecc4296b23924332c7df8bc from peer 146ef337cd9542698f96f62bed9a78ba ({username='slave'} at 127.21.58.70:43701)
I20240502 14:17:10.236721 4456 tablet_copy_service.cc:161] P 8f899bd3e9a148f5923c0277de1b3551: Beginning new tablet copy session on tablet 6d51ca94fecc4296b23924332c7df8bc from peer 146ef337cd9542698f96f62bed9a78ba at {username='slave'} at 127.21.58.70:43701: session id = 146ef337cd9542698f96f62bed9a78ba-6d51ca94fecc4296b23924332c7df8bc
I20240502 14:17:10.240687 4456 tablet_copy_source_session.cc:205] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.242558 5851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d51ca94fecc4296b23924332c7df8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.251024 5851 tablet_copy_client.cc:783] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.251431 5851 tablet_copy_client.cc:647] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.253309 5851 tablet_copy_client.cc:523] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.254226 5857 log.cc:824] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:10.257176 5851 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: Bootstrap starting.
I20240502 14:17:10.258850 5887 ts_tablet_manager.cc:902] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Initiating tablet copy from peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:10.281324 5859 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.281757 5859 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap complete.
I20240502 14:17:10.282059 5859 ts_tablet_manager.cc:1367] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: Time spent bootstrapping tablet: real 0.113s user 0.061s sys 0.007s
I20240502 14:17:10.283164 5859 raft_consensus.cc:348] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:10.283438 5859 raft_consensus.cc:729] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42b2fb8410fe4df48fb3d243f813c322, State: Initialized, Role: LEARNER
I20240502 14:17:10.283641 5859 consensus_queue.cc:260] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:10.284180 5859 ts_tablet_manager.cc:1398] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:17:10.286645 4324 tablet_copy_service.cc:342] P 751046160dd4466eb3f9ffea4128cdc5: Request end of tablet copy session 42b2fb8410fe4df48fb3d243f813c322-e295810c85204770b925390874f46d04 received from {username='slave'} at 127.21.58.68:41699
I20240502 14:17:10.286798 4324 tablet_copy_service.cc:433] P 751046160dd4466eb3f9ffea4128cdc5: ending tablet copy session 42b2fb8410fe4df48fb3d243f813c322-e295810c85204770b925390874f46d04 on tablet e295810c85204770b925390874f46d04 with peer 42b2fb8410fe4df48fb3d243f813c322
I20240502 14:17:10.302738 5887 tablet_copy_client.cc:308] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:41803
I20240502 14:17:10.303779 4192 tablet_copy_service.cc:140] P 79041fe14ee64478b7601e9b2230ccff: Received BeginTabletCopySession request for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 from peer cddf58ccc0c64875a1fb935bf72412ea ({username='slave'} at 127.21.58.69:50797)
I20240502 14:17:10.304039 4192 tablet_copy_service.cc:161] P 79041fe14ee64478b7601e9b2230ccff: Beginning new tablet copy session on tablet 00a9469b9baa42f9868fbff0b4d9f0a4 from peer cddf58ccc0c64875a1fb935bf72412ea at {username='slave'} at 127.21.58.69:50797: session id = cddf58ccc0c64875a1fb935bf72412ea-00a9469b9baa42f9868fbff0b4d9f0a4
I20240502 14:17:10.308224 4192 tablet_copy_source_session.cc:205] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:10.309823 5887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a9469b9baa42f9868fbff0b4d9f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:10.321511 5887 tablet_copy_client.cc:783] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:10.321833 5887 tablet_copy_client.cc:647] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:10.326135 5887 tablet_copy_client.cc:523] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:10.329833 5887 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap starting.
I20240502 14:17:10.344194 5851 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.344669 5851 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: Bootstrap complete.
I20240502 14:17:10.344949 5851 ts_tablet_manager.cc:1367] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: Time spent bootstrapping tablet: real 0.088s user 0.035s sys 0.022s
I20240502 14:17:10.345759 5851 raft_consensus.cc:348] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:10.346040 5851 raft_consensus.cc:729] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 146ef337cd9542698f96f62bed9a78ba, State: Initialized, Role: LEARNER
I20240502 14:17:10.346269 5851 consensus_queue.cc:260] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:10.347545 5851 ts_tablet_manager.cc:1398] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:17:10.348239 4456 tablet_copy_service.cc:342] P 8f899bd3e9a148f5923c0277de1b3551: Request end of tablet copy session 146ef337cd9542698f96f62bed9a78ba-6d51ca94fecc4296b23924332c7df8bc received from {username='slave'} at 127.21.58.70:43701
I20240502 14:17:10.348394 4456 tablet_copy_service.cc:433] P 8f899bd3e9a148f5923c0277de1b3551: ending tablet copy session 146ef337cd9542698f96f62bed9a78ba-6d51ca94fecc4296b23924332c7df8bc on tablet 6d51ca94fecc4296b23924332c7df8bc with peer 146ef337cd9542698f96f62bed9a78ba
I20240502 14:17:10.374644 5864 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.375275 5864 tablet_bootstrap.cc:492] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap complete.
I20240502 14:17:10.375821 5864 ts_tablet_manager.cc:1367] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: Time spent bootstrapping tablet: real 0.146s user 0.061s sys 0.006s
I20240502 14:17:10.379616 5857 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.380458 5857 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap complete.
I20240502 14:17:10.380789 5857 ts_tablet_manager.cc:1367] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: Time spent bootstrapping tablet: real 0.156s user 0.060s sys 0.007s
I20240502 14:17:10.383137 5864 raft_consensus.cc:348] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:10.383709 5864 raft_consensus.cc:729] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cddf58ccc0c64875a1fb935bf72412ea, State: Initialized, Role: LEARNER
I20240502 14:17:10.384313 5864 consensus_queue.cc:260] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:10.384889 5857 raft_consensus.cc:348] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:10.385108 5857 raft_consensus.cc:729] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cddf58ccc0c64875a1fb935bf72412ea, State: Initialized, Role: LEARNER
I20240502 14:17:10.385344 5857 consensus_queue.cc:260] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:10.389747 5362 raft_consensus.cc:1203] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.404912 5864 ts_tablet_manager.cc:1398] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea: Time spent starting tablet: real 0.029s user 0.009s sys 0.010s
I20240502 14:17:10.405603 5857 ts_tablet_manager.cc:1398] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea: Time spent starting tablet: real 0.025s user 0.010s sys 0.002s
I20240502 14:17:10.406615 5887 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:10.406844 4192 tablet_copy_service.cc:342] P 79041fe14ee64478b7601e9b2230ccff: Request end of tablet copy session cddf58ccc0c64875a1fb935bf72412ea-4149880ddb9d4e319c6386302d89fe0a received from {username='slave'} at 127.21.58.69:50797
I20240502 14:17:10.407001 4192 tablet_copy_service.cc:433] P 79041fe14ee64478b7601e9b2230ccff: ending tablet copy session cddf58ccc0c64875a1fb935bf72412ea-4149880ddb9d4e319c6386302d89fe0a on tablet 4149880ddb9d4e319c6386302d89fe0a with peer cddf58ccc0c64875a1fb935bf72412ea
I20240502 14:17:10.406960 4324 tablet_copy_service.cc:342] P 751046160dd4466eb3f9ffea4128cdc5: Request end of tablet copy session cddf58ccc0c64875a1fb935bf72412ea-efe680d0451a49f4b20da6ae2032223a received from {username='slave'} at 127.21.58.69:33207
I20240502 14:17:10.407164 5887 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap complete.
I20240502 14:17:10.407157 4324 tablet_copy_service.cc:433] P 751046160dd4466eb3f9ffea4128cdc5: ending tablet copy session cddf58ccc0c64875a1fb935bf72412ea-efe680d0451a49f4b20da6ae2032223a on tablet efe680d0451a49f4b20da6ae2032223a with peer cddf58ccc0c64875a1fb935bf72412ea
I20240502 14:17:10.407474 5887 ts_tablet_manager.cc:1367] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Time spent bootstrapping tablet: real 0.078s user 0.058s sys 0.009s
I20240502 14:17:10.408444 5887 raft_consensus.cc:348] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:10.409040 5887 raft_consensus.cc:729] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cddf58ccc0c64875a1fb935bf72412ea, State: Initialized, Role: LEARNER
I20240502 14:17:10.409307 5887 consensus_queue.cc:260] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:10.410285 5887 ts_tablet_manager.cc:1398] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:17:10.411095 4192 tablet_copy_service.cc:342] P 79041fe14ee64478b7601e9b2230ccff: Request end of tablet copy session cddf58ccc0c64875a1fb935bf72412ea-00a9469b9baa42f9868fbff0b4d9f0a4 received from {username='slave'} at 127.21.58.69:50797
I20240502 14:17:10.411226 4192 tablet_copy_service.cc:433] P 79041fe14ee64478b7601e9b2230ccff: ending tablet copy session cddf58ccc0c64875a1fb935bf72412ea-00a9469b9baa42f9868fbff0b4d9f0a4 on tablet 00a9469b9baa42f9868fbff0b4d9f0a4 with peer cddf58ccc0c64875a1fb935bf72412ea
I20240502 14:17:10.444149 5231 raft_consensus.cc:1203] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.517557 5231 raft_consensus.cc:1203] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.519498 5362 raft_consensus.cc:1203] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.594462 5492 raft_consensus.cc:1203] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.633719 5492 raft_consensus.cc:1203] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.796145 5231 raft_consensus.cc:1203] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.815897 5492 raft_consensus.cc:1203] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:10.897349 5928 raft_consensus.cc:1050] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: attempting to promote NON_VOTER 146ef337cd9542698f96f62bed9a78ba to VOTER
I20240502 14:17:10.898204 5928 consensus_queue.cc:237] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:10.900573 4172 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.900609 4304 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.900646 5231 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.901171 5928 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:10.901535 5980 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:10.901839 4978 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:10.909785 5928 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.910431 4172 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.910904 5231 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.914325 4304 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.916425 4027 catalog_manager.cc:5484] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.923815 4434 consensus_queue.cc:237] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:10.925544 5231 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:10.925741 4172 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:10.926277 5928 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:10.926698 5980 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:10.928614 5928 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.929291 4172 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.930781 5231 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:10.932757 4015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 326ac01b1c6c4107bb9b682ddb5af56f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:10.935302 4022 catalog_manager.cc:5484] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:10.940801 4283 tablet_service.cc:1514] Processing DeleteTablet for tablet 326ac01b1c6c4107bb9b682ddb5af56f with delete_type TABLET_DATA_TOMBSTONED (TS 751046160dd4466eb3f9ffea4128cdc5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50912
I20240502 14:17:10.941648 5993 tablet_replica.cc:330] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: stopping tablet replica
I20240502 14:17:10.942101 5993 raft_consensus.cc:2229] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:10.942552 5993 raft_consensus.cc:2258] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:10.943104 5911 raft_consensus.cc:1050] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: attempting to promote NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 to VOTER
I20240502 14:17:10.944036 5911 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:10.944272 5993 ts_tablet_manager.cc:1875] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:10.950404 5993 ts_tablet_manager.cc:1888] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:10.950624 5993 log.cc:1196] T 326ac01b1c6c4107bb9b682ddb5af56f P 751046160dd4466eb3f9ffea4128cdc5: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/326ac01b1c6c4107bb9b682ddb5af56f
I20240502 14:17:10.951503 4018 catalog_manager.cc:4830] TS 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867): tablet 326ac01b1c6c4107bb9b682ddb5af56f (table rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]) successfully deleted
I20240502 14:17:10.956467 5362 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.956678 4434 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.957304 4172 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.957334 5741 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:10.957937 5906 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:10.958284 5911 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 14:17:10.962613 5911 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:10.964215 4434 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:10.969775 4172 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:10.970654 4029 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:10.972335 5362 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:10.988106 4304 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:10.992102 4434 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:10.992610 5906 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:10.995039 5362 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:10.995599 5906 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:10.996382 5911 raft_consensus.cc:1050] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: attempting to promote NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 to VOTER
I20240502 14:17:10.997155 5911 consensus_queue.cc:237] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:10.998489 5741 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:10.999647 4435 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.002568 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e295810c85204770b925390874f46d04 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:11.004578 4022 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:11.009704 5980 raft_consensus.cc:1050] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551: attempting to promote NON_VOTER 146ef337cd9542698f96f62bed9a78ba to VOTER
I20240502 14:17:11.010776 5980 consensus_queue.cc:237] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:11.013967 4434 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.014773 6011 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.017629 4151 tablet_service.cc:1514] Processing DeleteTablet for tablet e295810c85204770b925390874f46d04 with delete_type TABLET_DATA_TOMBSTONED (TS 79041fe14ee64478b7601e9b2230ccff not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60774
I20240502 14:17:11.018545 6019 tablet_replica.cc:330] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: stopping tablet replica
I20240502 14:17:11.020033 6019 raft_consensus.cc:2229] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:11.021101 6019 raft_consensus.cc:2258] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:11.020541 4172 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.022384 5906 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.022722 5362 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.023758 5362 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.029400 5231 raft_consensus.cc:1261] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.030231 5880 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.032305 5906 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.034588 6019 ts_tablet_manager.cc:1875] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:11.038149 4304 raft_consensus.cc:1261] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.038913 5880 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.038903 5911 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.039902 4171 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.045055 4434 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.045605 5362 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.048271 4022 catalog_manager.cc:5484] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:11.049750 6019 ts_tablet_manager.cc:1888] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:11.049950 6019 log.cc:1196] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/e295810c85204770b925390874f46d04
I20240502 14:17:11.051175 4018 catalog_manager.cc:4830] TS 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): tablet e295810c85204770b925390874f46d04 (table rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]) successfully deleted
I20240502 14:17:11.058195 4304 consensus_queue.cc:237] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:11.060680 5362 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.060976 4434 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.061305 6013 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.061667 6015 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.059995 4171 raft_consensus.cc:1261] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.064500 5880 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.065701 5911 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.066828 4435 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.070089 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ec22af56f7144fdbe5fc8f361af3555 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:11.070442 5362 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:11.071458 4023 catalog_manager.cc:5484] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 5 to 6, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:11.076655 5880 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.080116 4151 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec22af56f7144fdbe5fc8f361af3555 with delete_type TABLET_DATA_TOMBSTONED (TS 79041fe14ee64478b7601e9b2230ccff not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60774
I20240502 14:17:11.080511 6019 tablet_replica.cc:330] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: stopping tablet replica
I20240502 14:17:11.080776 6019 raft_consensus.cc:2229] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:11.080633 5231 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.081174 6019 raft_consensus.cc:2258] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:11.081573 4304 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.082340 6019 ts_tablet_manager.cc:1875] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:11.086181 4172 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.085871 4023 catalog_manager.cc:5484] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba reported cstate change: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.090055 6019 ts_tablet_manager.cc:1888] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:11.090190 6019 log.cc:1196] T 5ec22af56f7144fdbe5fc8f361af3555 P 79041fe14ee64478b7601e9b2230ccff: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/5ec22af56f7144fdbe5fc8f361af3555
I20240502 14:17:11.091308 4018 catalog_manager.cc:4830] TS 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): tablet 5ec22af56f7144fdbe5fc8f361af3555 (table rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]) successfully deleted
I20240502 14:17:11.092247 4435 consensus_queue.cc:237] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:11.095278 5231 raft_consensus.cc:1261] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.095865 5880 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.096650 4172 raft_consensus.cc:1261] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.097199 5980 consensus_queue.cc:1035] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.098232 5880 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.100698 4171 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.101838 5230 raft_consensus.cc:2932] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.105703 4023 catalog_manager.cc:5484] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.108188 4015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:11.109655 4283 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 with delete_type TABLET_DATA_TOMBSTONED (TS 751046160dd4466eb3f9ffea4128cdc5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50912
I20240502 14:17:11.110000 5993 tablet_replica.cc:330] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: stopping tablet replica
I20240502 14:17:11.110181 5993 raft_consensus.cc:2229] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:11.110395 5993 raft_consensus.cc:2258] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:11.111274 5993 ts_tablet_manager.cc:1875] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:11.116554 5993 ts_tablet_manager.cc:1888] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:11.116677 5993 log.cc:1196] T ecd6d3a24c3f47bfa0dbf306d787b4a0 P 751046160dd4466eb3f9ffea4128cdc5: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/ecd6d3a24c3f47bfa0dbf306d787b4a0
I20240502 14:17:11.117437 4018 catalog_manager.cc:4830] TS 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867): tablet ecd6d3a24c3f47bfa0dbf306d787b4a0 (table rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]) successfully deleted
I20240502 14:17:11.167291 5741 raft_consensus.cc:1050] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5: attempting to promote NON_VOTER cddf58ccc0c64875a1fb935bf72412ea to VOTER
I20240502 14:17:11.168466 5741 consensus_queue.cc:237] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:11.171708 5492 raft_consensus.cc:1261] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.172505 6014 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.176961 4435 raft_consensus.cc:1261] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.177704 4171 raft_consensus.cc:1261] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.178299 5741 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.178623 6015 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.182348 5741 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.183135 4435 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.186693 4171 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.188406 4029 catalog_manager.cc:5484] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.192098 5492 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.194538 4304 consensus_queue.cc:237] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:11.196609 4172 raft_consensus.cc:1261] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.197108 5492 raft_consensus.cc:1261] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.197616 6014 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.198173 5741 consensus_queue.cc:1035] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.199879 6014 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.200502 4171 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.204039 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efe680d0451a49f4b20da6ae2032223a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:11.204062 5491 raft_consensus.cc:2932] T efe680d0451a49f4b20da6ae2032223a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.204939 4029 catalog_manager.cc:5484] T efe680d0451a49f4b20da6ae2032223a P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:11.209241 5938 raft_consensus.cc:1050] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: attempting to promote NON_VOTER cddf58ccc0c64875a1fb935bf72412ea to VOTER
I20240502 14:17:11.210116 5938 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:11.210701 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet efe680d0451a49f4b20da6ae2032223a with delete_type TABLET_DATA_TOMBSTONED (TS 8f899bd3e9a148f5923c0277de1b3551 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59962
I20240502 14:17:11.211925 6077 tablet_replica.cc:330] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: stopping tablet replica
I20240502 14:17:11.213843 5713 raft_consensus.cc:1050] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: attempting to promote NON_VOTER cddf58ccc0c64875a1fb935bf72412ea to VOTER
I20240502 14:17:11.213941 5492 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.214566 5713 consensus_queue.cc:237] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:11.214756 4435 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.215381 6077 raft_consensus.cc:2229] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:11.215926 6077 raft_consensus.cc:2258] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:11.216146 4304 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.216830 5937 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.217257 6082 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.217861 5937 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.218439 6077 ts_tablet_manager.cc:1875] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:11.220278 4434 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.220608 5491 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.220831 5937 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.221329 5937 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.221956 4303 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.229980 6083 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.252914 5713 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.252846 6080 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.255527 4302 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.257146 5491 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.257761 6077 ts_tablet_manager.cc:1888] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:11.257937 6077 log.cc:1196] T efe680d0451a49f4b20da6ae2032223a P 8f899bd3e9a148f5923c0277de1b3551: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/efe680d0451a49f4b20da6ae2032223a
I20240502 14:17:11.258797 4015 catalog_manager.cc:4830] TS 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879): tablet efe680d0451a49f4b20da6ae2032223a (table rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]) successfully deleted
I20240502 14:17:11.260429 4029 catalog_manager.cc:5484] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:11.260565 4435 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.260815 4432 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.261139 5492 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.264500 4022 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.264014 4303 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.270169 4171 consensus_queue.cc:237] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:11.270169 4172 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:11.274439 5492 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.275763 4302 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.277570 5491 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.278046 6082 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.278586 6082 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.279716 6082 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.281545 5937 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.284237 4303 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.284628 5713 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.285254 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4149880ddb9d4e319c6386302d89fe0a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:11.285562 5937 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.287309 5490 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.287638 4303 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.289222 4029 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:11.290974 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:11.294154 5492 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.294983 4302 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:11.302635 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 4149880ddb9d4e319c6386302d89fe0a with delete_type TABLET_DATA_TOMBSTONED (TS 8f899bd3e9a148f5923c0277de1b3551 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59962
I20240502 14:17:11.303130 6077 tablet_replica.cc:330] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: stopping tablet replica
I20240502 14:17:11.303321 6077 raft_consensus.cc:2229] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:11.303624 6077 raft_consensus.cc:2258] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:11.304562 4023 catalog_manager.cc:5484] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 5 to 6, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:11.304945 6077 ts_tablet_manager.cc:1875] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:11.308706 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 with delete_type TABLET_DATA_TOMBSTONED (TS 8f899bd3e9a148f5923c0277de1b3551 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59962
I20240502 14:17:11.311141 6077 ts_tablet_manager.cc:1888] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:11.311306 6077 log.cc:1196] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/4149880ddb9d4e319c6386302d89fe0a
I20240502 14:17:11.312083 6077 tablet_replica.cc:330] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: stopping tablet replica
I20240502 14:17:11.312155 4015 catalog_manager.cc:4830] TS 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879): tablet 4149880ddb9d4e319c6386302d89fe0a (table rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]) successfully deleted
I20240502 14:17:11.312387 6077 raft_consensus.cc:2229] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:11.312724 6077 raft_consensus.cc:2258] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:11.313467 6077 ts_tablet_manager.cc:1875] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:11.318822 6077 ts_tablet_manager.cc:1888] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:11.318943 6077 log.cc:1196] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 8f899bd3e9a148f5923c0277de1b3551: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/00a9469b9baa42f9868fbff0b4d9f0a4
I20240502 14:17:11.319698 4015 catalog_manager.cc:4830] TS 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879): tablet 00a9469b9baa42f9868fbff0b4d9f0a4 (table rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]) successfully deleted
I20240502 14:17:11.383862 5880 raft_consensus.cc:1050] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: attempting to promote NON_VOTER 146ef337cd9542698f96f62bed9a78ba to VOTER
I20240502 14:17:11.384716 5880 consensus_queue.cc:237] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:11.386929 4172 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.386987 4303 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.387001 5231 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 LEARNER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:11.387552 5880 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.387874 5928 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.388141 5980 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:11.392422 5880 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.395360 4303 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.396836 4172 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.400800 5231 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.401844 4029 catalog_manager.cc:5484] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 4 to 5, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.408619 4434 consensus_queue.cc:237] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:11.410059 5231 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.410161 4172 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:11.410575 5880 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.410871 5980 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:11.412875 5880 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.413508 4172 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.416525 4015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d51ca94fecc4296b23924332c7df8bc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:11.416817 5231 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:11.417647 4027 catalog_manager.cc:5484] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 5 to 6, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:11.422369 4283 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d51ca94fecc4296b23924332c7df8bc with delete_type TABLET_DATA_TOMBSTONED (TS 751046160dd4466eb3f9ffea4128cdc5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50912
I20240502 14:17:11.422709 5993 tablet_replica.cc:330] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: stopping tablet replica
I20240502 14:17:11.422888 5993 raft_consensus.cc:2229] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:11.423264 5993 raft_consensus.cc:2258] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:11.424028 5993 ts_tablet_manager.cc:1875] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:11.428149 5993 ts_tablet_manager.cc:1888] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:11.428256 5993 log.cc:1196] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d51ca94fecc4296b23924332c7df8bc
I20240502 14:17:11.428884 4018 catalog_manager.cc:4830] TS 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867): tablet 6d51ca94fecc4296b23924332c7df8bc (table rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]) successfully deleted
I20240502 14:17:11.640417 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:11.641381 5342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:11.668259 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:11.685652 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:11.686007 4151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:11.694236 5472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:11.754703 4172 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:11.757102 4303 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:11.757251 4434 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:11.757634 6083 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:11.757989 5938 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:11.760851 5713 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:11.761526 4434 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
W20240502 14:17:11.765813 4106 consensus_peers.cc:485] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff -> Peer 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023): Couldn't send request to peer 42b2fb8410fe4df48fb3d243f813c322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ebb0e689b4541f9867ae823046f7844. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:11.766697 4303 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:11.767868 4022 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index -1 to 4, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:12.267119 6178 ts_tablet_manager.cc:902] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Initiating tablet copy from peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:12.268517 6178 tablet_copy_client.cc:308] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:41803
I20240502 14:17:12.273998 4192 tablet_copy_service.cc:140] P 79041fe14ee64478b7601e9b2230ccff: Received BeginTabletCopySession request for tablet 9ebb0e689b4541f9867ae823046f7844 from peer 42b2fb8410fe4df48fb3d243f813c322 ({username='slave'} at 127.21.58.68:40821)
I20240502 14:17:12.274174 4192 tablet_copy_service.cc:161] P 79041fe14ee64478b7601e9b2230ccff: Beginning new tablet copy session on tablet 9ebb0e689b4541f9867ae823046f7844 from peer 42b2fb8410fe4df48fb3d243f813c322 at {username='slave'} at 127.21.58.68:40821: session id = 42b2fb8410fe4df48fb3d243f813c322-9ebb0e689b4541f9867ae823046f7844
I20240502 14:17:12.277127 4192 tablet_copy_source_session.cc:205] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:12.278332 6178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0e689b4541f9867ae823046f7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:12.282644 6178 tablet_copy_client.cc:783] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:12.282874 6178 tablet_copy_client.cc:647] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:12.284497 6178 tablet_copy_client.cc:523] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:12.287914 6178 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap starting.
I20240502 14:17:12.318370 6178 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:12.318677 6178 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap complete.
I20240502 14:17:12.318883 6178 ts_tablet_manager.cc:1367] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s
I20240502 14:17:12.319572 6178 raft_consensus.cc:348] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:12.319765 6178 raft_consensus.cc:729] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42b2fb8410fe4df48fb3d243f813c322, State: Initialized, Role: LEARNER
I20240502 14:17:12.319916 6178 consensus_queue.cc:260] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:12.320545 6178 ts_tablet_manager.cc:1398] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240502 14:17:12.321223 4192 tablet_copy_service.cc:342] P 79041fe14ee64478b7601e9b2230ccff: Request end of tablet copy session 42b2fb8410fe4df48fb3d243f813c322-9ebb0e689b4541f9867ae823046f7844 received from {username='slave'} at 127.21.58.68:40821
I20240502 14:17:12.321370 4192 tablet_copy_service.cc:433] P 79041fe14ee64478b7601e9b2230ccff: ending tablet copy session 42b2fb8410fe4df48fb3d243f813c322-9ebb0e689b4541f9867ae823046f7844 on tablet 9ebb0e689b4541f9867ae823046f7844 with peer 42b2fb8410fe4df48fb3d243f813c322
I20240502 14:17:12.690851 5362 raft_consensus.cc:1203] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:13.123582 6197 raft_consensus.cc:1050] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: attempting to promote NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 to VOTER
I20240502 14:17:13.124236 6197 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:13.126161 4434 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:13.126292 5362 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:13.126343 4303 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:13.126778 6197 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:13.127130 6210 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:13.127442 6198 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:13.130365 6198 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:13.131371 4434 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:13.131675 5362 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:13.134207 4303 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:13.136487 4022 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 4 to 5, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:13.250934 4172 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844"
dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41170
I20240502 14:17:13.251222 4172 raft_consensus.cc:595] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:13.468573 4172 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844"
dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41192
I20240502 14:17:13.468791 4172 raft_consensus.cc:595] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:13.516559 6197 raft_consensus.cc:979] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: : Instructing follower 751046160dd4466eb3f9ffea4128cdc5 to start an election
I20240502 14:17:13.516762 6197 raft_consensus.cc:1067] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Signalling peer 751046160dd4466eb3f9ffea4128cdc5 to start an election
I20240502 14:17:13.517563 4303 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844"
dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
from {username='slave'} at 127.21.58.65:50859
I20240502 14:17:13.517822 4303 raft_consensus.cc:482] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:13.517936 4303 raft_consensus.cc:3037] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:13.519963 4303 raft_consensus.cc:504] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:13.520638 4303 leader_election.cc:288] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 2 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879), 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023)
I20240502 14:17:13.521018 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:13.521159 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:13.521262 4172 raft_consensus.cc:3032] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:13.521353 4172 raft_consensus.cc:729] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Running, Role: LEADER
I20240502 14:17:13.521373 4434 raft_consensus.cc:3037] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:13.521489 5362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0e689b4541f9867ae823046f7844" candidate_uuid: "751046160dd4466eb3f9ffea4128cdc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "42b2fb8410fe4df48fb3d243f813c322"
I20240502 14:17:13.521660 4172 consensus_queue.cc:260] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:13.521826 5362 raft_consensus.cc:3037] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:13.522092 4172 raft_consensus.cc:3037] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:13.524142 4434 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 2.
I20240502 14:17:13.524773 5362 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 2.
I20240502 14:17:13.525048 4172 raft_consensus.cc:2454] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751046160dd4466eb3f9ffea4128cdc5 in term 2.
I20240502 14:17:13.525218 4238 leader_election.cc:302] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 42b2fb8410fe4df48fb3d243f813c322, 751046160dd4466eb3f9ffea4128cdc5, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:13.525534 5906 raft_consensus.cc:2781] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:13.525933 5906 raft_consensus.cc:686] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Becoming Leader. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Running, Role: LEADER
I20240502 14:17:13.526219 5906 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:13.528867 4029 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: term changed from 1 to 2, leader changed from 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) to 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66). New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:14.057554 5362 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:14.058096 5906 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:14.124622 4434 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:14.125092 5906 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:14.129436 4172 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:14.130125 6259 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:14.133716 4303 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:14.135442 5362 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:14.135485 4434 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:14.136042 6256 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.136448 6256 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.138068 5906 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:14.138759 4434 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:14.140798 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ebb0e689b4541f9867ae823046f7844 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:14.142136 4027 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 5 to 7, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:14.142917 5362 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:14.146955 4151 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebb0e689b4541f9867ae823046f7844 with delete_type TABLET_DATA_TOMBSTONED (TS 79041fe14ee64478b7601e9b2230ccff not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60774
I20240502 14:17:14.147743 6265 tablet_replica.cc:330] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: stopping tablet replica
I20240502 14:17:14.147913 6265 raft_consensus.cc:2229] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:14.148087 6265 raft_consensus.cc:2258] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:14.148775 6265 ts_tablet_manager.cc:1875] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:14.152850 6265 ts_tablet_manager.cc:1888] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:17:14.152951 6265 log.cc:1196] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ebb0e689b4541f9867ae823046f7844
I20240502 14:17:14.153594 4018 catalog_manager.cc:4830] TS 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): tablet 9ebb0e689b4541f9867ae823046f7844 (table rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]) successfully deleted
I20240502 14:17:14.392916 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:14.408497 5342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:14.411295 4283 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:17:14.425823 4151 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:17:14.430694 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:14.441824 5472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:14.601620 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:14.609555 4283 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:17:14.613767 5342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:14.639048 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:14.640225 5472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:14.643767 4151 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:17:14.720944 4303 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } }
I20240502 14:17:14.722011 4302 consensus_queue.cc:237] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:14.723327 5362 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 14:17:14.723667 4240 consensus_peers.cc:485] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): Couldn't send request to peer 79041fe14ee64478b7601e9b2230ccff. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.725070 4432 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:14.725075 4434 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.725190 5362 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:17:14.725497 4238 consensus_peers.cc:485] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69:37277): Couldn't send request to peer cddf58ccc0c64875a1fb935bf72412ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ec22af56f7144fdbe5fc8f361af3555. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.725870 6358 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.725519 6356 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 14:17:14.726284 6359 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.726495 5906 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:14.729665 6356 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } }
I20240502 14:17:14.729928 4434 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } }
I20240502 14:17:14.730432 5906 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:14.731055 4432 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:14.735371 4027 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 7 to 8, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:14.738976 5362 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } }
I20240502 14:17:14.750564 5361 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:14.767560 4023 catalog_manager.cc:5484] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 6 to 7, NON_VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } } }
I20240502 14:17:14.784489 4434 consensus_queue.cc:237] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:14.784492 4302 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } }
W20240502 14:17:14.787127 4240 consensus_peers.cc:485] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): Couldn't send request to peer 79041fe14ee64478b7601e9b2230ccff. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.787313 4432 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.787459 5361 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.787863 6356 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.788398 6358 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.789321 5231 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.789947 6362 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.791633 4172 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.792456 6362 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.802277 6358 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } }
I20240502 14:17:14.806931 4434 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } }
I20240502 14:17:14.809549 4029 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 6 to 7, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:17:14.810135 4369 consensus_peers.cc:485] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 -> Peer 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023): Couldn't send request to peer 42b2fb8410fe4df48fb3d243f813c322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 326ac01b1c6c4107bb9b682ddb5af56f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.808204 5361 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } }
I20240502 14:17:14.821805 5880 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:14.822875 4172 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:14.830626 4029 catalog_manager.cc:5484] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:14.839004 5230 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:14.850292 4172 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: true } }
I20240502 14:17:14.853704 5491 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:17:14.853701 4105 consensus_peers.cc:485] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff -> Peer 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879): Couldn't send request to peer 8f899bd3e9a148f5923c0277de1b3551. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.853787 4302 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.854254 6399 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.854784 6197 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.856320 4172 consensus_queue.cc:237] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:14.857937 6399 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: true } } }
I20240502 14:17:14.859743 4303 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.860036 5492 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:17:14.860713 4106 consensus_peers.cc:485] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff -> Peer 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023): Couldn't send request to peer 42b2fb8410fe4df48fb3d243f813c322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00a9469b9baa42f9868fbff0b4d9f0a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.860450 4302 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: true } } }
I20240502 14:17:14.861018 6197 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.861534 5492 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: true } } }
I20240502 14:17:14.863163 6400 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.866802 4027 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 6 to 7, NON_VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) added. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:14.874372 6197 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:14.880831 4303 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:14.879227 5492 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } } }
I20240502 14:17:14.882040 4029 catalog_manager.cc:5484] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 6 to 7, NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:14.891563 4434 consensus_queue.cc:237] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: true } }
I20240502 14:17:14.894174 4172 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:17:14.894568 4371 consensus_peers.cc:485] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 -> Peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867): Couldn't send request to peer 751046160dd4466eb3f9ffea4128cdc5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.894596 5230 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:14.894893 6388 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.895222 5880 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:14.898039 5880 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: true } } }
I20240502 14:17:14.898643 5230 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: true } } }
I20240502 14:17:14.899047 4172 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: true } } }
I20240502 14:17:14.905505 4029 catalog_manager.cc:5484] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 6 to 7, NON_VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) added. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:14.915959 4303 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:14.918907 5361 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:14.918907 4434 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 14:17:14.919492 4240 consensus_peers.cc:485] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): Couldn't send request to peer 79041fe14ee64478b7601e9b2230ccff. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.919570 6359 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:14.920095 6359 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:14.923298 5906 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:14.928011 4434 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
W20240502 14:17:14.928803 4237 consensus_peers.cc:485] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer 146ef337cd9542698f96f62bed9a78ba (127.21.58.70:44877): Couldn't send request to peer 146ef337cd9542698f96f62bed9a78ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ebb0e689b4541f9867ae823046f7844. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:14.930186 5361 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:14.933102 4022 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 8 to 9, NON_VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) added. New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:15.233654 6490 ts_tablet_manager.cc:902] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Initiating tablet copy from peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:15.236416 6490 tablet_copy_client.cc:308] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:32867
I20240502 14:17:15.238974 4324 tablet_copy_service.cc:140] P 751046160dd4466eb3f9ffea4128cdc5: Received BeginTabletCopySession request for tablet e295810c85204770b925390874f46d04 from peer 79041fe14ee64478b7601e9b2230ccff ({username='slave'} at 127.21.58.65:50859)
I20240502 14:17:15.239197 4324 tablet_copy_service.cc:161] P 751046160dd4466eb3f9ffea4128cdc5: Beginning new tablet copy session on tablet e295810c85204770b925390874f46d04 from peer 79041fe14ee64478b7601e9b2230ccff at {username='slave'} at 127.21.58.65:50859: session id = 79041fe14ee64478b7601e9b2230ccff-e295810c85204770b925390874f46d04
I20240502 14:17:15.242125 4324 tablet_copy_source_session.cc:205] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.244065 6490 ts_tablet_manager.cc:1875] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:17:15.251242 6490 ts_tablet_manager.cc:1888] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:17:15.251580 6490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e295810c85204770b925390874f46d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.253734 6490 tablet_copy_client.cc:783] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.254014 6490 tablet_copy_client.cc:647] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.256129 6490 tablet_copy_client.cc:523] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.259276 6490 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:15.300828 6490 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.301295 6490 tablet_bootstrap.cc:492] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap complete.
I20240502 14:17:15.301550 6490 ts_tablet_manager.cc:1367] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.042s user 0.035s sys 0.005s
I20240502 14:17:15.302178 6490 raft_consensus.cc:348] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } }
I20240502 14:17:15.302382 6490 raft_consensus.cc:729] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: LEARNER
I20240502 14:17:15.302599 6490 consensus_queue.cc:260] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } }
I20240502 14:17:15.303207 6490 ts_tablet_manager.cc:1398] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:17:15.303974 4324 tablet_copy_service.cc:342] P 751046160dd4466eb3f9ffea4128cdc5: Request end of tablet copy session 79041fe14ee64478b7601e9b2230ccff-e295810c85204770b925390874f46d04 received from {username='slave'} at 127.21.58.65:50859
I20240502 14:17:15.304136 4324 tablet_copy_service.cc:433] P 751046160dd4466eb3f9ffea4128cdc5: ending tablet copy session 79041fe14ee64478b7601e9b2230ccff-e295810c85204770b925390874f46d04 on tablet e295810c85204770b925390874f46d04 with peer 79041fe14ee64478b7601e9b2230ccff
I20240502 14:17:15.317715 6510 ts_tablet_manager.cc:902] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Initiating tablet copy from peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:15.319360 6510 tablet_copy_client.cc:308] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:41803
I20240502 14:17:15.319984 4192 tablet_copy_service.cc:140] P 79041fe14ee64478b7601e9b2230ccff: Received BeginTabletCopySession request for tablet 4149880ddb9d4e319c6386302d89fe0a from peer 8f899bd3e9a148f5923c0277de1b3551 ({username='slave'} at 127.21.58.67:34963)
I20240502 14:17:15.320153 4192 tablet_copy_service.cc:161] P 79041fe14ee64478b7601e9b2230ccff: Beginning new tablet copy session on tablet 4149880ddb9d4e319c6386302d89fe0a from peer 8f899bd3e9a148f5923c0277de1b3551 at {username='slave'} at 127.21.58.67:34963: session id = 8f899bd3e9a148f5923c0277de1b3551-4149880ddb9d4e319c6386302d89fe0a
I20240502 14:17:15.322050 4192 tablet_copy_source_session.cc:205] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.323264 6510 ts_tablet_manager.cc:1875] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:17:15.327551 6510 ts_tablet_manager.cc:1888] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:17:15.327811 6510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4149880ddb9d4e319c6386302d89fe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.330276 6515 ts_tablet_manager.cc:902] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: Initiating tablet copy from peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:15.330369 6510 tablet_copy_client.cc:783] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.330660 6510 tablet_copy_client.cc:647] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.332566 6515 tablet_copy_client.cc:308] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:32867
I20240502 14:17:15.333139 6510 tablet_copy_client.cc:523] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.333349 4324 tablet_copy_service.cc:140] P 751046160dd4466eb3f9ffea4128cdc5: Received BeginTabletCopySession request for tablet 5ec22af56f7144fdbe5fc8f361af3555 from peer cddf58ccc0c64875a1fb935bf72412ea ({username='slave'} at 127.21.58.69:33207)
I20240502 14:17:15.333510 4324 tablet_copy_service.cc:161] P 751046160dd4466eb3f9ffea4128cdc5: Beginning new tablet copy session on tablet 5ec22af56f7144fdbe5fc8f361af3555 from peer cddf58ccc0c64875a1fb935bf72412ea at {username='slave'} at 127.21.58.69:33207: session id = cddf58ccc0c64875a1fb935bf72412ea-5ec22af56f7144fdbe5fc8f361af3555
I20240502 14:17:15.335681 4324 tablet_copy_source_session.cc:205] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.335924 6510 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap starting.
I20240502 14:17:15.336794 6515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec22af56f7144fdbe5fc8f361af3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.337622 6490 ts_tablet_manager.cc:902] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Initiating tablet copy from peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:15.338959 6490 tablet_copy_client.cc:308] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:32867
I20240502 14:17:15.339648 4324 tablet_copy_service.cc:140] P 751046160dd4466eb3f9ffea4128cdc5: Received BeginTabletCopySession request for tablet 9ebb0e689b4541f9867ae823046f7844 from peer 79041fe14ee64478b7601e9b2230ccff ({username='slave'} at 127.21.58.65:50859)
I20240502 14:17:15.339815 4324 tablet_copy_service.cc:161] P 751046160dd4466eb3f9ffea4128cdc5: Beginning new tablet copy session on tablet 9ebb0e689b4541f9867ae823046f7844 from peer 79041fe14ee64478b7601e9b2230ccff at {username='slave'} at 127.21.58.65:50859: session id = 79041fe14ee64478b7601e9b2230ccff-9ebb0e689b4541f9867ae823046f7844
I20240502 14:17:15.342672 4324 tablet_copy_source_session.cc:205] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.342717 6515 tablet_copy_client.cc:783] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.342960 6515 tablet_copy_client.cc:647] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.344380 6490 ts_tablet_manager.cc:1875] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:17:15.344583 6515 tablet_copy_client.cc:523] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.346987 6515 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap starting.
I20240502 14:17:15.350775 6490 ts_tablet_manager.cc:1888] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 14:17:15.351032 6490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0e689b4541f9867ae823046f7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.354149 6490 tablet_copy_client.cc:783] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.354349 6490 tablet_copy_client.cc:647] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.355434 6526 ts_tablet_manager.cc:902] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: Initiating tablet copy from peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:15.356463 6490 tablet_copy_client.cc:523] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.359673 6490 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap starting.
I20240502 14:17:15.387499 6531 ts_tablet_manager.cc:902] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: Initiating tablet copy from peer 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:15.389086 6526 tablet_copy_client.cc:308] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:41803
I20240502 14:17:15.389823 4192 tablet_copy_service.cc:140] P 79041fe14ee64478b7601e9b2230ccff: Received BeginTabletCopySession request for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 from peer 42b2fb8410fe4df48fb3d243f813c322 ({username='slave'} at 127.21.58.68:40821)
I20240502 14:17:15.389990 4192 tablet_copy_service.cc:161] P 79041fe14ee64478b7601e9b2230ccff: Beginning new tablet copy session on tablet 00a9469b9baa42f9868fbff0b4d9f0a4 from peer 42b2fb8410fe4df48fb3d243f813c322 at {username='slave'} at 127.21.58.68:40821: session id = 42b2fb8410fe4df48fb3d243f813c322-00a9469b9baa42f9868fbff0b4d9f0a4
I20240502 14:17:15.391250 6510 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.391855 6510 tablet_bootstrap.cc:492] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Bootstrap complete.
I20240502 14:17:15.392215 6510 ts_tablet_manager.cc:1367] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Time spent bootstrapping tablet: real 0.056s user 0.047s sys 0.008s
I20240502 14:17:15.392853 4192 tablet_copy_source_session.cc:205] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.393170 6510 raft_consensus.cc:348] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: true } }
I20240502 14:17:15.393519 6510 raft_consensus.cc:729] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f899bd3e9a148f5923c0277de1b3551, State: Initialized, Role: LEARNER
I20240502 14:17:15.393779 6510 consensus_queue.cc:260] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: NON_VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: true } }
I20240502 14:17:15.394511 6510 ts_tablet_manager.cc:1398] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:17:15.395274 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a9469b9baa42f9868fbff0b4d9f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.395259 4192 tablet_copy_service.cc:342] P 79041fe14ee64478b7601e9b2230ccff: Request end of tablet copy session 8f899bd3e9a148f5923c0277de1b3551-4149880ddb9d4e319c6386302d89fe0a received from {username='slave'} at 127.21.58.67:34963
I20240502 14:17:15.395439 4192 tablet_copy_service.cc:433] P 79041fe14ee64478b7601e9b2230ccff: ending tablet copy session 8f899bd3e9a148f5923c0277de1b3551-4149880ddb9d4e319c6386302d89fe0a on tablet 4149880ddb9d4e319c6386302d89fe0a with peer 8f899bd3e9a148f5923c0277de1b3551
I20240502 14:17:15.405222 6515 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.405696 6515 tablet_bootstrap.cc:492] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: Bootstrap complete.
I20240502 14:17:15.406008 6515 ts_tablet_manager.cc:1367] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: Time spent bootstrapping tablet: real 0.059s user 0.051s sys 0.004s
I20240502 14:17:15.406121 6531 tablet_copy_client.cc:308] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:39879
I20240502 14:17:15.406754 6534 ts_tablet_manager.cc:902] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: Initiating tablet copy from peer 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867)
I20240502 14:17:15.406819 6515 raft_consensus.cc:348] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:15.407130 6515 raft_consensus.cc:729] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cddf58ccc0c64875a1fb935bf72412ea, State: Initialized, Role: LEARNER
I20240502 14:17:15.407310 6526 tablet_copy_client.cc:783] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.407337 6515 consensus_queue.cc:260] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: true } }
I20240502 14:17:15.407568 6526 tablet_copy_client.cc:647] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.409595 6526 tablet_copy_client.cc:523] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.411561 6515 ts_tablet_manager.cc:1398] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240502 14:17:15.412554 4324 tablet_copy_service.cc:342] P 751046160dd4466eb3f9ffea4128cdc5: Request end of tablet copy session cddf58ccc0c64875a1fb935bf72412ea-5ec22af56f7144fdbe5fc8f361af3555 received from {username='slave'} at 127.21.58.69:33207
I20240502 14:17:15.412680 4324 tablet_copy_service.cc:433] P 751046160dd4466eb3f9ffea4128cdc5: ending tablet copy session cddf58ccc0c64875a1fb935bf72412ea-5ec22af56f7144fdbe5fc8f361af3555 on tablet 5ec22af56f7144fdbe5fc8f361af3555 with peer cddf58ccc0c64875a1fb935bf72412ea
I20240502 14:17:15.413369 6526 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap starting.
I20240502 14:17:15.419814 4456 tablet_copy_service.cc:140] P 8f899bd3e9a148f5923c0277de1b3551: Received BeginTabletCopySession request for tablet 326ac01b1c6c4107bb9b682ddb5af56f from peer 42b2fb8410fe4df48fb3d243f813c322 ({username='slave'} at 127.21.58.68:51905)
I20240502 14:17:15.419986 4456 tablet_copy_service.cc:161] P 8f899bd3e9a148f5923c0277de1b3551: Beginning new tablet copy session on tablet 326ac01b1c6c4107bb9b682ddb5af56f from peer 42b2fb8410fe4df48fb3d243f813c322 at {username='slave'} at 127.21.58.68:51905: session id = 42b2fb8410fe4df48fb3d243f813c322-326ac01b1c6c4107bb9b682ddb5af56f
I20240502 14:17:15.422860 4456 tablet_copy_source_session.cc:205] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.424423 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326ac01b1c6c4107bb9b682ddb5af56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.427171 6534 tablet_copy_client.cc:308] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:32867
I20240502 14:17:15.431054 6531 tablet_copy_client.cc:783] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.431311 6531 tablet_copy_client.cc:647] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.433105 6531 tablet_copy_client.cc:523] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.436493 6531 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap starting.
I20240502 14:17:15.458315 6490 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.458904 6490 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Bootstrap complete.
I20240502 14:17:15.459911 4324 tablet_copy_service.cc:140] P 751046160dd4466eb3f9ffea4128cdc5: Received BeginTabletCopySession request for tablet 9ebb0e689b4541f9867ae823046f7844 from peer 146ef337cd9542698f96f62bed9a78ba ({username='slave'} at 127.21.58.70:56283)
I20240502 14:17:15.460131 4324 tablet_copy_service.cc:161] P 751046160dd4466eb3f9ffea4128cdc5: Beginning new tablet copy session on tablet 9ebb0e689b4541f9867ae823046f7844 from peer 146ef337cd9542698f96f62bed9a78ba at {username='slave'} at 127.21.58.70:56283: session id = 146ef337cd9542698f96f62bed9a78ba-9ebb0e689b4541f9867ae823046f7844
I20240502 14:17:15.461349 6490 ts_tablet_manager.cc:1367] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Time spent bootstrapping tablet: real 0.102s user 0.049s sys 0.014s
I20240502 14:17:15.462450 6490 raft_consensus.cc:348] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:15.462826 6490 raft_consensus.cc:729] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Initialized, Role: LEARNER
I20240502 14:17:15.463109 6490 consensus_queue.cc:260] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:15.464524 4324 tablet_copy_source_session.cc:205] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.466049 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0e689b4541f9867ae823046f7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.467681 6490 ts_tablet_manager.cc:1398] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:17:15.468508 4324 tablet_copy_service.cc:342] P 751046160dd4466eb3f9ffea4128cdc5: Request end of tablet copy session 79041fe14ee64478b7601e9b2230ccff-9ebb0e689b4541f9867ae823046f7844 received from {username='slave'} at 127.21.58.65:50859
I20240502 14:17:15.468696 4324 tablet_copy_service.cc:433] P 751046160dd4466eb3f9ffea4128cdc5: ending tablet copy session 79041fe14ee64478b7601e9b2230ccff-9ebb0e689b4541f9867ae823046f7844 on tablet 9ebb0e689b4541f9867ae823046f7844 with peer 79041fe14ee64478b7601e9b2230ccff
I20240502 14:17:15.475800 6534 tablet_copy_client.cc:783] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.476053 6534 tablet_copy_client.cc:647] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.477710 6534 tablet_copy_client.cc:523] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.481357 6534 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: Bootstrap starting.
I20240502 14:17:15.507954 6554 ts_tablet_manager.cc:902] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Initiating tablet copy from peer 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:15.526437 6526 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.526926 6526 tablet_bootstrap.cc:492] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap complete.
I20240502 14:17:15.527240 6526 ts_tablet_manager.cc:1367] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: Time spent bootstrapping tablet: real 0.114s user 0.051s sys 0.003s
I20240502 14:17:15.528074 6526 raft_consensus.cc:348] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:15.528371 6526 raft_consensus.cc:729] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42b2fb8410fe4df48fb3d243f813c322, State: Initialized, Role: LEARNER
I20240502 14:17:15.528596 6526 consensus_queue.cc:260] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:15.530079 6554 tablet_copy_client.cc:308] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:39879
I20240502 14:17:15.530980 4456 tablet_copy_service.cc:140] P 8f899bd3e9a148f5923c0277de1b3551: Received BeginTabletCopySession request for tablet 6d51ca94fecc4296b23924332c7df8bc from peer 751046160dd4466eb3f9ffea4128cdc5 ({username='slave'} at 127.21.58.66:43149)
I20240502 14:17:15.531162 4456 tablet_copy_service.cc:161] P 8f899bd3e9a148f5923c0277de1b3551: Beginning new tablet copy session on tablet 6d51ca94fecc4296b23924332c7df8bc from peer 751046160dd4466eb3f9ffea4128cdc5 at {username='slave'} at 127.21.58.66:43149: session id = 751046160dd4466eb3f9ffea4128cdc5-6d51ca94fecc4296b23924332c7df8bc
I20240502 14:17:15.534152 4456 tablet_copy_source_session.cc:205] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:15.535815 6554 ts_tablet_manager.cc:1875] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:17:15.542943 6554 ts_tablet_manager.cc:1888] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:17:15.543273 6554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d51ca94fecc4296b23924332c7df8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:15.550709 6526 ts_tablet_manager.cc:1398] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20240502 14:17:15.551379 4192 tablet_copy_service.cc:342] P 79041fe14ee64478b7601e9b2230ccff: Request end of tablet copy session 42b2fb8410fe4df48fb3d243f813c322-00a9469b9baa42f9868fbff0b4d9f0a4 received from {username='slave'} at 127.21.58.68:40821
I20240502 14:17:15.551534 4192 tablet_copy_service.cc:433] P 79041fe14ee64478b7601e9b2230ccff: ending tablet copy session 42b2fb8410fe4df48fb3d243f813c322-00a9469b9baa42f9868fbff0b4d9f0a4 on tablet 00a9469b9baa42f9868fbff0b4d9f0a4 with peer 42b2fb8410fe4df48fb3d243f813c322
I20240502 14:17:15.553005 6554 tablet_copy_client.cc:783] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:15.553303 6554 tablet_copy_client.cc:647] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:15.555123 6531 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.555608 6531 tablet_bootstrap.cc:492] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: Bootstrap complete.
I20240502 14:17:15.555945 6531 ts_tablet_manager.cc:1367] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: Time spent bootstrapping tablet: real 0.120s user 0.068s sys 0.024s
I20240502 14:17:15.556136 6554 tablet_copy_client.cc:523] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:15.556780 6531 raft_consensus.cc:348] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:15.557080 6531 raft_consensus.cc:729] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42b2fb8410fe4df48fb3d243f813c322, State: Initialized, Role: LEARNER
I20240502 14:17:15.557309 6531 consensus_queue.cc:260] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: true } }
I20240502 14:17:15.557873 6531 ts_tablet_manager.cc:1398] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:17:15.558501 4456 tablet_copy_service.cc:342] P 8f899bd3e9a148f5923c0277de1b3551: Request end of tablet copy session 42b2fb8410fe4df48fb3d243f813c322-326ac01b1c6c4107bb9b682ddb5af56f received from {username='slave'} at 127.21.58.68:51905
I20240502 14:17:15.558660 4456 tablet_copy_service.cc:433] P 8f899bd3e9a148f5923c0277de1b3551: ending tablet copy session 42b2fb8410fe4df48fb3d243f813c322-326ac01b1c6c4107bb9b682ddb5af56f on tablet 326ac01b1c6c4107bb9b682ddb5af56f with peer 42b2fb8410fe4df48fb3d243f813c322
I20240502 14:17:15.558820 6534 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.559341 6534 tablet_bootstrap.cc:492] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: Bootstrap complete.
I20240502 14:17:15.559684 6534 ts_tablet_manager.cc:1367] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: Time spent bootstrapping tablet: real 0.078s user 0.057s sys 0.006s
I20240502 14:17:15.560549 6534 raft_consensus.cc:348] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:15.560966 6534 raft_consensus.cc:729] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 146ef337cd9542698f96f62bed9a78ba, State: Initialized, Role: LEARNER
I20240502 14:17:15.561246 6534 consensus_queue.cc:260] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: NON_VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: true } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:15.567440 6554 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap starting.
I20240502 14:17:15.575656 6534 ts_tablet_manager.cc:1398] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20240502 14:17:15.576483 4324 tablet_copy_service.cc:342] P 751046160dd4466eb3f9ffea4128cdc5: Request end of tablet copy session 146ef337cd9542698f96f62bed9a78ba-9ebb0e689b4541f9867ae823046f7844 received from {username='slave'} at 127.21.58.70:56283
I20240502 14:17:15.576618 4324 tablet_copy_service.cc:433] P 751046160dd4466eb3f9ffea4128cdc5: ending tablet copy session 146ef337cd9542698f96f62bed9a78ba-9ebb0e689b4541f9867ae823046f7844 on tablet 9ebb0e689b4541f9867ae823046f7844 with peer 146ef337cd9542698f96f62bed9a78ba
I20240502 14:17:15.637794 6554 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:15.638478 6554 tablet_bootstrap.cc:492] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Bootstrap complete.
I20240502 14:17:15.638798 6554 ts_tablet_manager.cc:1367] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Time spent bootstrapping tablet: real 0.072s user 0.054s sys 0.006s
I20240502 14:17:15.639581 6554 raft_consensus.cc:348] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: true } }
I20240502 14:17:15.639922 6554 raft_consensus.cc:729] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Initialized, Role: LEARNER
I20240502 14:17:15.640156 6554 consensus_queue.cc:260] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: NON_VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: true } }
I20240502 14:17:15.640924 6554 ts_tablet_manager.cc:1398] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:17:15.641297 4172 raft_consensus.cc:1203] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:17:15.642254 4456 tablet_copy_service.cc:342] P 8f899bd3e9a148f5923c0277de1b3551: Request end of tablet copy session 751046160dd4466eb3f9ffea4128cdc5-6d51ca94fecc4296b23924332c7df8bc received from {username='slave'} at 127.21.58.66:43149
I20240502 14:17:15.642434 4456 tablet_copy_service.cc:433] P 8f899bd3e9a148f5923c0277de1b3551: ending tablet copy session 751046160dd4466eb3f9ffea4128cdc5-6d51ca94fecc4296b23924332c7df8bc on tablet 6d51ca94fecc4296b23924332c7df8bc with peer 751046160dd4466eb3f9ffea4128cdc5
I20240502 14:17:15.731317 4434 raft_consensus.cc:1203] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:17:15.808272 4172 raft_consensus.cc:1203] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 14:17:15.811056 5361 raft_consensus.cc:1203] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:17:15.881004 5492 raft_consensus.cc:1203] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:17:15.908792 4303 raft_consensus.cc:1203] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:17:15.954648 5230 raft_consensus.cc:1203] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 14:17:15.977165 5361 raft_consensus.cc:1203] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:17:16.098155 6598 raft_consensus.cc:1050] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: attempting to promote NON_VOTER 79041fe14ee64478b7601e9b2230ccff to VOTER
I20240502 14:17:16.098925 6598 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } }
I20240502 14:17:16.101579 4172 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.101780 5361 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.102210 4434 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.102197 6600 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.102818 6599 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.103420 6600 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.107666 6600 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:16.108335 4172 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:16.108436 4434 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:16.114953 4022 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 7 to 8, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:16.117043 5361 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:16.117630 6602 raft_consensus.cc:1050] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: attempting to promote NON_VOTER 8f899bd3e9a148f5923c0277de1b3551 to VOTER
I20240502 14:17:16.118726 6602 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } }
I20240502 14:17:16.128545 5492 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.129271 6601 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.129773 4434 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEARNER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.129994 4303 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.130254 6601 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.130733 6601 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.137017 6602 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:16.137814 4303 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:16.143226 4434 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:16.143646 4027 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 7 to 8, 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:16.158690 5492 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:16.183972 4303 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e295810c85204770b925390874f46d04"
dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56894
I20240502 14:17:16.184165 4303 raft_consensus.cc:595] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:16.195787 6601 raft_consensus.cc:1050] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff: attempting to promote NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 to VOTER
I20240502 14:17:16.196427 6601 consensus_queue.cc:237] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:16.198276 5361 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.198300 4303 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.198417 5492 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.198774 6400 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.199146 6677 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.199388 6602 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.212194 6677 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.212978 4303 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.215680 5361 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.217563 5492 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.218207 4024 catalog_manager.cc:5484] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.223850 4172 consensus_queue.cc:237] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:16.225499 5361 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.225898 4303 raft_consensus.cc:1261] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 79041fe14ee64478b7601e9b2230ccff: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.226408 6677 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.226730 6601 consensus_queue.cc:1035] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.229362 6602 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.230096 5361 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.230068 4303 raft_consensus.cc:2932] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.232249 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:16.233201 4024 catalog_manager.cc:5484] T 00a9469b9baa42f9868fbff0b4d9f0a4 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 8 to 9, VOTER cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) evicted. New cstate: current_term: 1 leader_uuid: "79041fe14ee64478b7601e9b2230ccff" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:16.244779 5472 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a9469b9baa42f9868fbff0b4d9f0a4 with delete_type TABLET_DATA_TOMBSTONED (TS cddf58ccc0c64875a1fb935bf72412ea not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43792
I20240502 14:17:16.245597 6709 tablet_replica.cc:330] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: stopping tablet replica
I20240502 14:17:16.246017 6709 raft_consensus.cc:2229] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:16.246362 6709 raft_consensus.cc:2258] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:16.247543 6709 ts_tablet_manager.cc:1875] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:16.251890 6709 ts_tablet_manager.cc:1888] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:17:16.252028 6709 log.cc:1196] T 00a9469b9baa42f9868fbff0b4d9f0a4 P cddf58ccc0c64875a1fb935bf72412ea: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/wals/00a9469b9baa42f9868fbff0b4d9f0a4
I20240502 14:17:16.252748 4016 catalog_manager.cc:4830] TS cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69:37277): tablet 00a9469b9baa42f9868fbff0b4d9f0a4 (table rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]) successfully deleted
I20240502 14:17:16.368257 6600 raft_consensus.cc:1050] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: attempting to promote NON_VOTER 79041fe14ee64478b7601e9b2230ccff to VOTER
I20240502 14:17:16.368969 6600 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:16.371320 4434 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:16.371523 5230 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:16.371807 4172 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:16.371927 5361 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:16.372094 6599 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:16.372345 6711 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:16.372699 6358 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:16.372911 6600 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:16.392369 6599 raft_consensus.cc:1013] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: attempt to promote peer 146ef337cd9542698f96f62bed9a78ba: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:17:16.393329 6711 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.393880 4172 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.399559 5361 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.399840 4029 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 9 to 10, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240502 14:17:16.403424 6599 raft_consensus.cc:1050] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5: attempting to promote NON_VOTER cddf58ccc0c64875a1fb935bf72412ea to VOTER
I20240502 14:17:16.404079 5230 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.404569 6599 consensus_queue.cc:237] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:16.394644 4434 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.411338 4303 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } }
I20240502 14:17:16.413686 5362 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:17:16.414291 4172 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:17:16.414440 6711 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:17:16.415731 6358 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:17:16.417263 6600 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.417495 5362 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.420696 4172 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.422925 4029 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 reported cstate change: config changed from index 10 to 11, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.424368 5231 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:17:16.425115 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ebb0e689b4541f9867ae823046f7844 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:16.425242 6358 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:17:16.429817 5231 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: NON_VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: true } } }
I20240502 14:17:16.433905 6358 raft_consensus.cc:1050] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5: attempting to promote NON_VOTER 146ef337cd9542698f96f62bed9a78ba to VOTER
I20240502 14:17:16.435001 6358 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:16.435959 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebb0e689b4541f9867ae823046f7844 with delete_type TABLET_DATA_TOMBSTONED (TS 8f899bd3e9a148f5923c0277de1b3551 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59962
I20240502 14:17:16.437542 5361 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:17:16.437625 5230 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:17:16.438764 4172 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:17:16.439497 6600 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:17:16.439810 6598 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:17:16.440281 6598 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:17:16.441268 6643 raft_consensus.cc:1050] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551: attempting to promote NON_VOTER 42b2fb8410fe4df48fb3d243f813c322 to VOTER
I20240502 14:17:16.442117 6643 consensus_queue.cc:237] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:16.444340 6644 raft_consensus.cc:1050] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551: attempting to promote NON_VOTER 751046160dd4466eb3f9ffea4128cdc5 to VOTER
I20240502 14:17:16.445168 6644 consensus_queue.cc:237] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } }
I20240502 14:17:16.445971 6358 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.446621 5361 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.451915 4172 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.452963 5492 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [term 1 LEARNER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.453552 6598 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.456779 5230 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.457288 4029 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 11 to 12, 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.463116 4303 consensus_queue.cc:237] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } }
I20240502 14:17:16.464089 5362 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.464428 6728 tablet_replica.cc:330] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: stopping tablet replica
I20240502 14:17:16.464676 6728 raft_consensus.cc:2229] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:16.464872 6358 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.465507 4172 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:17:16.466127 6598 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:17:16.466789 5230 raft_consensus.cc:1261] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:17:16.467821 6728 raft_consensus.cc:2258] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:16.468025 5362 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [term 1 LEARNER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.468264 6598 consensus_queue.cc:1035] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:17:16.468549 5880 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.468983 6728 ts_tablet_manager.cc:1875] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:16.469130 6358 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.469641 4172 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.473853 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ebb0e689b4541f9867ae823046f7844 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:16.474032 5230 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.474090 4171 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.474622 5880 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.475212 5880 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.476230 4023 catalog_manager.cc:5484] T 9ebb0e689b4541f9867ae823046f7844 P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 12 to 13, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New cstate: current_term: 2 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.477100 5231 raft_consensus.cc:2932] T 9ebb0e689b4541f9867ae823046f7844 P 146ef337cd9542698f96f62bed9a78ba [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } }
I20240502 14:17:16.480324 6728 ts_tablet_manager.cc:1888] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 14:17:16.481168 6728 log.cc:1196] T 9ebb0e689b4541f9867ae823046f7844 P 8f899bd3e9a148f5923c0277de1b3551: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/9ebb0e689b4541f9867ae823046f7844
I20240502 14:17:16.481313 4434 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.482997 6600 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.484107 4015 catalog_manager.cc:4830] TS 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879): tablet 9ebb0e689b4541f9867ae823046f7844 (table rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]) successfully deleted
I20240502 14:17:16.493144 5231 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.493314 4303 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEARNER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.493036 6600 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.493917 5880 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.494462 6643 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.497131 5360 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.505016 4172 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:16.507287 4027 catalog_manager.cc:5484] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 reported cstate change: config changed from index 7 to 8, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:16.507701 5492 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.508287 6643 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:16.515863 6644 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.516556 5361 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.521225 4171 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.522245 4023 catalog_manager.cc:5484] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 reported cstate change: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.523339 5342 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebb0e689b4541f9867ae823046f7844 with delete_type TABLET_DATA_TOMBSTONED (TS 42b2fb8410fe4df48fb3d243f813c322 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:54036
I20240502 14:17:16.531118 6600 raft_consensus.cc:979] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: : Instructing follower 42b2fb8410fe4df48fb3d243f813c322 to start an election
I20240502 14:17:16.531306 6600 raft_consensus.cc:1067] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Signalling peer 42b2fb8410fe4df48fb3d243f813c322 to start an election
I20240502 14:17:16.532836 5362 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e295810c85204770b925390874f46d04"
dest_uuid: "42b2fb8410fe4df48fb3d243f813c322"
from {username='slave'} at 127.21.58.66:60279
I20240502 14:17:16.533082 5362 raft_consensus.cc:482] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:16.533202 5362 raft_consensus.cc:3037] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.533329 4302 consensus_queue.cc:237] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } }
I20240502 14:17:16.535815 5362 raft_consensus.cc:504] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } }
I20240502 14:17:16.535696 4432 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false replace: true } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.538367 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e295810c85204770b925390874f46d04" candidate_uuid: "42b2fb8410fe4df48fb3d243f813c322" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:16.538591 4435 raft_consensus.cc:3037] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.538769 5362 leader_election.cc:288] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [CANDIDATE]: Term 2 election: Requested vote from peers 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879), 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803)
I20240502 14:17:16.539273 4171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e295810c85204770b925390874f46d04" candidate_uuid: "42b2fb8410fe4df48fb3d243f813c322" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:16.539526 4171 raft_consensus.cc:3037] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.539948 5492 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.540683 6358 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.542428 4171 raft_consensus.cc:2454] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b2fb8410fe4df48fb3d243f813c322 in term 2.
I20240502 14:17:16.543555 6358 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.545881 6749 tablet_replica.cc:330] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: stopping tablet replica
I20240502 14:17:16.546371 6749 raft_consensus.cc:2229] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:16.546420 4435 raft_consensus.cc:2454] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b2fb8410fe4df48fb3d243f813c322 in term 2.
I20240502 14:17:16.546839 6749 raft_consensus.cc:2258] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:16.547153 5295 leader_election.cc:302] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 42b2fb8410fe4df48fb3d243f813c322, 79041fe14ee64478b7601e9b2230ccff, 8f899bd3e9a148f5923c0277de1b3551; no voters:
I20240502 14:17:16.547622 6564 raft_consensus.cc:2781] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:16.548363 4302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e295810c85204770b925390874f46d04" candidate_uuid: "42b2fb8410fe4df48fb3d243f813c322" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:16.548573 4302 raft_consensus.cc:3032] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:16.548688 4302 raft_consensus.cc:729] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 751046160dd4466eb3f9ffea4128cdc5, State: Running, Role: LEADER
I20240502 14:17:16.549026 4302 consensus_queue.cc:260] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } }
I20240502 14:17:16.549520 4302 raft_consensus.cc:3037] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.550122 5492 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.551352 4435 raft_consensus.cc:1261] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Refusing update from remote peer 751046160dd4466eb3f9ffea4128cdc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.552049 6749 ts_tablet_manager.cc:1875] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:16.552073 6600 consensus_queue.cc:1035] T 5ec22af56f7144fdbe5fc8f361af3555 P 751046160dd4466eb3f9ffea4128cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.553273 5230 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false replace: true } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.553575 4435 raft_consensus.cc:2932] T 5ec22af56f7144fdbe5fc8f361af3555 P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.554539 6643 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.555292 4029 catalog_manager.cc:5484] T 5ec22af56f7144fdbe5fc8f361af3555 P cddf58ccc0c64875a1fb935bf72412ea reported cstate change: config changed from index 8 to 9, VOTER 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68) evicted. New cstate: current_term: 1 leader_uuid: "751046160dd4466eb3f9ffea4128cdc5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } }
I20240502 14:17:16.560333 5342 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec22af56f7144fdbe5fc8f361af3555 with delete_type TABLET_DATA_TOMBSTONED (TS 42b2fb8410fe4df48fb3d243f813c322 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54036
I20240502 14:17:16.562362 4302 raft_consensus.cc:2454] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b2fb8410fe4df48fb3d243f813c322 in term 2.
I20240502 14:17:16.562291 4172 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.563810 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ec22af56f7144fdbe5fc8f361af3555 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:16.565896 6749 ts_tablet_manager.cc:1888] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240502 14:17:16.566113 6749 log.cc:1196] T 9ebb0e689b4541f9867ae823046f7844 P 42b2fb8410fe4df48fb3d243f813c322: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ebb0e689b4541f9867ae823046f7844
I20240502 14:17:16.567509 4303 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.568450 6749 tablet_replica.cc:330] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: stopping tablet replica
I20240502 14:17:16.568452 4016 catalog_manager.cc:4830] TS 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023): tablet 9ebb0e689b4541f9867ae823046f7844 (table rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]) successfully deleted
I20240502 14:17:16.568480 4023 catalog_manager.cc:5484] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff reported cstate change: config changed from index 7 to 8, 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.568900 6749 raft_consensus.cc:2229] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:16.569187 6749 raft_consensus.cc:2258] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:16.570741 4434 consensus_queue.cc:237] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } }
I20240502 14:17:16.571875 6749 ts_tablet_manager.cc:1875] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:16.580888 5362 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.580886 4172 raft_consensus.cc:1261] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.581429 6644 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.582851 5880 consensus_queue.cc:1035] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.583760 6644 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.585115 5230 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.586927 4434 consensus_queue.cc:237] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } }
I20240502 14:17:16.588363 5362 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 42b2fb8410fe4df48fb3d243f813c322 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.589260 4015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 326ac01b1c6c4107bb9b682ddb5af56f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:16.589390 4303 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.589977 6564 raft_consensus.cc:686] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 LEADER]: Becoming Leader. State: Replica: 42b2fb8410fe4df48fb3d243f813c322, State: Running, Role: LEADER
I20240502 14:17:16.590104 5880 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.590504 6564 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } }
I20240502 14:17:16.592056 4172 raft_consensus.cc:2932] T 326ac01b1c6c4107bb9b682ddb5af56f P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } }
I20240502 14:17:16.592480 4023 catalog_manager.cc:5484] T 326ac01b1c6c4107bb9b682ddb5af56f P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 8 to 9, VOTER 146ef337cd9542698f96f62bed9a78ba (127.21.58.70) evicted. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:16.593127 6749 ts_tablet_manager.cc:1888] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:17:16.595808 6735 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.596468 6749 log.cc:1196] T 5ec22af56f7144fdbe5fc8f361af3555 P 42b2fb8410fe4df48fb3d243f813c322: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ec22af56f7144fdbe5fc8f361af3555
I20240502 14:17:16.597275 4016 catalog_manager.cc:4830] TS 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68:41023): tablet 5ec22af56f7144fdbe5fc8f361af3555 (table rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]) successfully deleted
I20240502 14:17:16.597151 4029 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 reported cstate change: term changed from 1 to 2, leader changed from 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) to 42b2fb8410fe4df48fb3d243f813c322 (127.21.58.68). New cstate: current_term: 2 leader_uuid: "42b2fb8410fe4df48fb3d243f813c322" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:16.600196 4015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d51ca94fecc4296b23924332c7df8bc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:16.603020 5231 raft_consensus.cc:1261] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Refusing update from remote peer 8f899bd3e9a148f5923c0277de1b3551: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:16.603719 6644 consensus_queue.cc:1035] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:16.603976 4303 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.606220 5231 raft_consensus.cc:2932] T 6d51ca94fecc4296b23924332c7df8bc P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } } }
I20240502 14:17:16.609741 4023 catalog_manager.cc:5484] T 6d51ca94fecc4296b23924332c7df8bc P 8f899bd3e9a148f5923c0277de1b3551 reported cstate change: config changed from index 8 to 9, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "8f899bd3e9a148f5923c0277de1b3551" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146ef337cd9542698f96f62bed9a78ba" member_type: VOTER last_known_addr { host: "127.21.58.70" port: 44877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:16.616091 4151 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d51ca94fecc4296b23924332c7df8bc with delete_type TABLET_DATA_TOMBSTONED (TS 79041fe14ee64478b7601e9b2230ccff not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60774
I20240502 14:17:16.620112 6757 tablet_replica.cc:330] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: stopping tablet replica
I20240502 14:17:16.620424 6757 raft_consensus.cc:2229] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:16.620752 6757 raft_consensus.cc:2258] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:16.621893 6757 ts_tablet_manager.cc:1875] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:16.630049 6757 ts_tablet_manager.cc:1888] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:17:16.630251 6757 log.cc:1196] T 6d51ca94fecc4296b23924332c7df8bc P 79041fe14ee64478b7601e9b2230ccff: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d51ca94fecc4296b23924332c7df8bc
I20240502 14:17:16.631150 4018 catalog_manager.cc:4830] TS 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): tablet 6d51ca94fecc4296b23924332c7df8bc (table rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]) successfully deleted
I20240502 14:17:16.631891 5211 tablet_service.cc:1514] Processing DeleteTablet for tablet 326ac01b1c6c4107bb9b682ddb5af56f with delete_type TABLET_DATA_TOMBSTONED (TS 146ef337cd9542698f96f62bed9a78ba not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54380
I20240502 14:17:16.637914 4172 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a"
dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42704
I20240502 14:17:16.638207 4172 raft_consensus.cc:595] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:16.645124 6764 tablet_replica.cc:330] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: stopping tablet replica
I20240502 14:17:16.645925 6764 raft_consensus.cc:2229] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:16.646672 6764 raft_consensus.cc:2258] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:16.649060 6764 ts_tablet_manager.cc:1875] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:16.657698 6764 ts_tablet_manager.cc:1888] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:17:16.657915 6764 log.cc:1196] T 326ac01b1c6c4107bb9b682ddb5af56f P 146ef337cd9542698f96f62bed9a78ba: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-5/wal/wals/326ac01b1c6c4107bb9b682ddb5af56f
I20240502 14:17:16.658818 4015 catalog_manager.cc:4830] TS 146ef337cd9542698f96f62bed9a78ba (127.21.58.70:44877): tablet 326ac01b1c6c4107bb9b682ddb5af56f (table rebalance_test_table_0 [id=b89b2895b59c412b89b659585aeac3a5]) successfully deleted
I20240502 14:17:16.698624 6601 raft_consensus.cc:979] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: : Instructing follower cddf58ccc0c64875a1fb935bf72412ea to start an election
I20240502 14:17:16.698772 6601 raft_consensus.cc:1067] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Signalling peer cddf58ccc0c64875a1fb935bf72412ea to start an election
I20240502 14:17:16.699544 5492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a"
dest_uuid: "cddf58ccc0c64875a1fb935bf72412ea"
from {username='slave'} at 127.21.58.65:35221
I20240502 14:17:16.699819 5492 raft_consensus.cc:482] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:16.700017 5492 raft_consensus.cc:3037] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.702946 5492 raft_consensus.cc:504] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } }
I20240502 14:17:16.705446 4172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a" candidate_uuid: "cddf58ccc0c64875a1fb935bf72412ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
I20240502 14:17:16.705742 4172 raft_consensus.cc:3032] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:16.705758 4303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a" candidate_uuid: "cddf58ccc0c64875a1fb935bf72412ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "751046160dd4466eb3f9ffea4128cdc5"
I20240502 14:17:16.705847 4172 raft_consensus.cc:729] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79041fe14ee64478b7601e9b2230ccff, State: Running, Role: LEADER
I20240502 14:17:16.705955 4303 raft_consensus.cc:3037] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.706240 4172 consensus_queue.cc:260] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } }
I20240502 14:17:16.706784 4172 raft_consensus.cc:3037] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.708834 4303 raft_consensus.cc:2454] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddf58ccc0c64875a1fb935bf72412ea in term 2.
I20240502 14:17:16.705495 5492 leader_election.cc:288] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [CANDIDATE]: Term 2 election: Requested vote from peers 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803), 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867), 8f899bd3e9a148f5923c0277de1b3551 (127.21.58.67:39879)
I20240502 14:17:16.710634 4172 raft_consensus.cc:2454] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddf58ccc0c64875a1fb935bf72412ea in term 2.
I20240502 14:17:16.712848 5428 leader_election.cc:302] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 751046160dd4466eb3f9ffea4128cdc5, 79041fe14ee64478b7601e9b2230ccff, cddf58ccc0c64875a1fb935bf72412ea; no voters:
I20240502 14:17:16.713325 6540 raft_consensus.cc:2781] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:16.715977 6540 raft_consensus.cc:686] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 2 LEADER]: Becoming Leader. State: Replica: cddf58ccc0c64875a1fb935bf72412ea, State: Running, Role: LEADER
I20240502 14:17:16.716557 6540 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } }
I20240502 14:17:16.721025 4023 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea reported cstate change: term changed from 1 to 2, leader changed from 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) to cddf58ccc0c64875a1fb935bf72412ea (127.21.58.69). New cstate: current_term: 2 leader_uuid: "cddf58ccc0c64875a1fb935bf72412ea" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:16.722357 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a" candidate_uuid: "cddf58ccc0c64875a1fb935bf72412ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8f899bd3e9a148f5923c0277de1b3551"
I20240502 14:17:16.722549 4434 raft_consensus.cc:3037] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:16.724617 4434 raft_consensus.cc:2454] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddf58ccc0c64875a1fb935bf72412ea in term 2.
I20240502 14:17:16.735702 4172 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4149880ddb9d4e319c6386302d89fe0a"
dest_uuid: "79041fe14ee64478b7601e9b2230ccff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42732
I20240502 14:17:16.735931 4172 raft_consensus.cc:595] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Received request to transfer leadership
I20240502 14:17:16.736030 4172 raft_consensus.cc:603] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 14:17:16.827328 5342 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:16.827849 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:16.840579 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:16.846621 5472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:16.853143 4151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:16.863458 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:17.046737 4171 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Refusing update from remote peer 42b2fb8410fe4df48fb3d243f813c322: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:17.047444 6751 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:17.111354 4434 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer 42b2fb8410fe4df48fb3d243f813c322: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:17.111932 6751 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:17.117753 4303 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Refusing update from remote peer 42b2fb8410fe4df48fb3d243f813c322: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:17.119307 6751 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:17.124788 5362 consensus_queue.cc:237] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } }
I20240502 14:17:17.128377 4171 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Refusing update from remote peer 42b2fb8410fe4df48fb3d243f813c322: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:17.128633 4434 raft_consensus.cc:1261] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer 42b2fb8410fe4df48fb3d243f813c322: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:17.129292 6859 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:17.129665 6861 consensus_queue.cc:1035] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:17.132068 6859 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:17.132669 4434 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:17.135391 4171 raft_consensus.cc:2932] T e295810c85204770b925390874f46d04 P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } } }
I20240502 14:17:17.137220 4016 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e295810c85204770b925390874f46d04 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:17.137471 4029 catalog_manager.cc:5484] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 reported cstate change: config changed from index 8 to 10, VOTER 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "42b2fb8410fe4df48fb3d243f813c322" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b2fb8410fe4df48fb3d243f813c322" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 41023 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:17.141557 4283 tablet_service.cc:1514] Processing DeleteTablet for tablet e295810c85204770b925390874f46d04 with delete_type TABLET_DATA_TOMBSTONED (TS 751046160dd4466eb3f9ffea4128cdc5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50912
I20240502 14:17:17.143198 6866 tablet_replica.cc:330] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: stopping tablet replica
I20240502 14:17:17.143500 6866 raft_consensus.cc:2229] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:17.143798 6866 raft_consensus.cc:2258] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:17.144903 6866 ts_tablet_manager.cc:1875] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:17.152546 6866 ts_tablet_manager.cc:1888] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 14:17:17.152722 6866 log.cc:1196] T e295810c85204770b925390874f46d04 P 751046160dd4466eb3f9ffea4128cdc5: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/e295810c85204770b925390874f46d04
I20240502 14:17:17.153631 4018 catalog_manager.cc:4830] TS 751046160dd4466eb3f9ffea4128cdc5 (127.21.58.66:32867): tablet e295810c85204770b925390874f46d04 (table rebalance_test_table_1 [id=ba9261892b5341c6988b06d2f7fd63b8]) successfully deleted
I20240502 14:17:17.210728 4171 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Refusing update from remote peer cddf58ccc0c64875a1fb935bf72412ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:17.211611 6540 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "79041fe14ee64478b7601e9b2230ccff" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 41803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:17.213141 4434 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer cddf58ccc0c64875a1fb935bf72412ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:17.214002 6540 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:17.223944 4303 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Refusing update from remote peer cddf58ccc0c64875a1fb935bf72412ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:17.224936 6874 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:17.810598 5492 consensus_queue.cc:237] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } }
I20240502 14:17:17.812266 4303 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Refusing update from remote peer cddf58ccc0c64875a1fb935bf72412ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:17.812426 4434 raft_consensus.cc:1261] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Refusing update from remote peer cddf58ccc0c64875a1fb935bf72412ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:17.812798 6897 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:17.813056 6898 consensus_queue.cc:1035] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:17.815250 6897 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:17.816069 4303 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 751046160dd4466eb3f9ffea4128cdc5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:17.816002 4434 raft_consensus.cc:2932] T 4149880ddb9d4e319c6386302d89fe0a P 8f899bd3e9a148f5923c0277de1b3551 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } } }
I20240502 14:17:17.819705 4016 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4149880ddb9d4e319c6386302d89fe0a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:17.820775 4029 catalog_manager.cc:5484] T 4149880ddb9d4e319c6386302d89fe0a P cddf58ccc0c64875a1fb935bf72412ea reported cstate change: config changed from index 8 to 10, VOTER 79041fe14ee64478b7601e9b2230ccff (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "cddf58ccc0c64875a1fb935bf72412ea" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "751046160dd4466eb3f9ffea4128cdc5" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddf58ccc0c64875a1fb935bf72412ea" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f899bd3e9a148f5923c0277de1b3551" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 39879 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:17.825215 4151 tablet_service.cc:1514] Processing DeleteTablet for tablet 4149880ddb9d4e319c6386302d89fe0a with delete_type TABLET_DATA_TOMBSTONED (TS 79041fe14ee64478b7601e9b2230ccff not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60774
I20240502 14:17:17.825809 6902 tablet_replica.cc:330] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: stopping tablet replica
I20240502 14:17:17.826005 6902 raft_consensus.cc:2229] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:17.826622 6902 raft_consensus.cc:2258] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:17.827581 6902 ts_tablet_manager.cc:1875] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:17.832016 6902 ts_tablet_manager.cc:1888] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 14:17:17.832120 6902 log.cc:1196] T 4149880ddb9d4e319c6386302d89fe0a P 79041fe14ee64478b7601e9b2230ccff: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/4149880ddb9d4e319c6386302d89fe0a
I20240502 14:17:17.832741 4018 catalog_manager.cc:4830] TS 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): tablet 4149880ddb9d4e319c6386302d89fe0a (table rebalance_test_table_2 [id=3ef754c394b24aa286281ecaab2e00ab]) successfully deleted
I20240502 14:17:18.002238 5342 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:18.008636 5472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:18.018687 4151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:18.036728 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:18.046543 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:18.051635 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:18.197528 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:18.207369 4283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:18.208297 5342 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:18.222343 5472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:18.223388 4151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:18.229574 4414 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:18.467825 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 4090
I20240502 14:17:18.475854 4211 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:18.626466 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 4090
W20240502 14:17:18.654906 4240 connection.cc:535] client connection to 127.21.58.65:41803 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:18.655196 4240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:17:18.655315 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 4222
W20240502 14:17:18.657004 4240 consensus_peers.cc:485] T 9ebb0e689b4541f9867ae823046f7844 P 751046160dd4466eb3f9ffea4128cdc5 -> Peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): Couldn't send request to peer 79041fe14ee64478b7601e9b2230ccff. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:41803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:18.661170 4343 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:17:18.675552 5298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.58.65:41803: connect: Connection refused (error 111)
W20240502 14:17:18.677048 5298 consensus_peers.cc:485] T e295810c85204770b925390874f46d04 P 42b2fb8410fe4df48fb3d243f813c322 -> Peer 79041fe14ee64478b7601e9b2230ccff (127.21.58.65:41803): Couldn't send request to peer 79041fe14ee64478b7601e9b2230ccff. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:41803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:18.822026 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 4222
I20240502 14:17:18.850847 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 4352
I20240502 14:17:18.857120 4476 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:19.017586 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 4352
I20240502 14:17:19.044440 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 5280
I20240502 14:17:19.050801 5402 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:19.173364 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 5280
W20240502 14:17:19.198196 4018 connection.cc:535] server connection from 127.21.58.68:34549 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:17:19.198485 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 5410
I20240502 14:17:19.204465 5532 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:19.321938 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 5410
I20240502 14:17:19.344563 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 5149
I20240502 14:17:19.350380 5271 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:19.467586 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 5149
I20240502 14:17:19.490088 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 4000
I20240502 14:17:19.491304 4061 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:19.638137 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 4000
2024-05-02T14:17:19Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (24234 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240502 14:17:19.698251 21737 test_util.cc:268] Using random seed: 1120952611
I20240502 14:17:19.700156 21737 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:17:19.700228 21737 ts_itest-base.cc:113] --------------
I20240502 14:17:19.700266 21737 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:17:19.700314 21737 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:17:19.700351 21737 ts_itest-base.cc:116] --------------
2024-05-02T14:17:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:17:19Z Disabled control of system clock
I20240502 14:17:19.744277 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:39077
--webserver_interface=127.21.58.126
--webserver_port=0
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.58.126:39077
--location_mapping_cmd=/tmp/dist-test-task3cgK4A/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 14:17:20.066306 7004 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 14:17:20.066630 7004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:20.066741 7004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:20.066845 7004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:20.074208 7004 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:20.074345 7004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:17:20.074451 7004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:20.074517 7004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:17:20.074604 7004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:17:20.083431 7004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task3cgK4A/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.21.58.126:39077
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.58.126:39077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.58.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:20.084625 7004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:20.086285 7004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:20.096446 7009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:20.097046 7012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:20.097622 7010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:20.097860 7004 server_base.cc:1034] running on GCE node
I20240502 14:17:20.098518 7004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:20.099845 7004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:20.101060 7004 hybrid_clock.cc:648] HybridClock initialized: now 1714659440101008 us; error 30 us; skew 500 ppm
I20240502 14:17:20.101346 7004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:20.103518 7004 webserver.cc:460] Webserver started at http://127.21.58.126:37755/ using document root <none> and password file <none>
I20240502 14:17:20.104241 7004 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:20.104370 7004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:20.104657 7004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:17:20.107283 7004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f0027bb3edc645d6a1ed389a8251ac34"
format_stamp: "Formatted at 2024-05-02 14:17:20 on dist-test-slave-wtgm"
I20240502 14:17:20.108062 7004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f0027bb3edc645d6a1ed389a8251ac34"
format_stamp: "Formatted at 2024-05-02 14:17:20 on dist-test-slave-wtgm"
I20240502 14:17:20.113008 7004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:17:20.116093 7018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:20.117249 7004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:17:20.117403 7004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f0027bb3edc645d6a1ed389a8251ac34"
format_stamp: "Formatted at 2024-05-02 14:17:20 on dist-test-slave-wtgm"
I20240502 14:17:20.117554 7004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:20.140100 7004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:20.140792 7004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:20.141098 7004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:20.157996 7004 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.126:39077
I20240502 14:17:20.158032 7070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.126:39077 every 8 connection(s)
I20240502 14:17:20.159214 7004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:17:20.162564 7071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:20.166376 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 7004
I20240502 14:17:20.166589 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:17:20.171782 7071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34: Bootstrap starting.
I20240502 14:17:20.175074 7071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:20.176378 7071 log.cc:824] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:20.178956 7071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34: No bootstrap required, opened a new log
I20240502 14:17:20.185194 7071 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0027bb3edc645d6a1ed389a8251ac34" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 39077 } }
I20240502 14:17:20.185497 7071 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:20.185590 7071 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0027bb3edc645d6a1ed389a8251ac34, State: Initialized, Role: FOLLOWER
I20240502 14:17:20.186196 7071 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0027bb3edc645d6a1ed389a8251ac34" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 39077 } }
I20240502 14:17:20.186417 7071 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:17:20.186545 7071 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:17:20.186697 7071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:20.188638 7071 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0027bb3edc645d6a1ed389a8251ac34" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 39077 } }
I20240502 14:17:20.189072 7071 leader_election.cc:302] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0027bb3edc645d6a1ed389a8251ac34; no voters:
I20240502 14:17:20.189604 7071 leader_election.cc:288] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:17:20.189716 7076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:20.190446 7076 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [term 1 LEADER]: Becoming Leader. State: Replica: f0027bb3edc645d6a1ed389a8251ac34, State: Running, Role: LEADER
I20240502 14:17:20.190982 7076 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0027bb3edc645d6a1ed389a8251ac34" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 39077 } }
I20240502 14:17:20.191381 7071 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:17:20.193547 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0027bb3edc645d6a1ed389a8251ac34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0027bb3edc645d6a1ed389a8251ac34" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 39077 } } }
I20240502 14:17:20.193795 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [sys.catalog]: This master's current role is: LEADER
I20240502 14:17:20.194404 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0027bb3edc645d6a1ed389a8251ac34. Latest consensus state: current_term: 1 leader_uuid: "f0027bb3edc645d6a1ed389a8251ac34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0027bb3edc645d6a1ed389a8251ac34" member_type: VOTER last_known_addr { host: "127.21.58.126" port: 39077 } } }
I20240502 14:17:20.194610 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34 [sys.catalog]: This master's current role is: LEADER
I20240502 14:17:20.195334 7082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:17:20.200671 7082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:17:20.209484 7082 catalog_manager.cc:1347] Generated new cluster ID: 0356f1d4f3934682b5f6379ec85613d8
I20240502 14:17:20.209580 7082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:17:20.223655 7082 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:17:20.224727 7082 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:17:20.235339 7082 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f0027bb3edc645d6a1ed389a8251ac34: Generated new TSK 0
I20240502 14:17:20.236104 7082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:17:20.248521 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.65:0
--local_ip_for_outbound_sockets=127.21.58.65
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:17:20.599673 7095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:20.599941 7095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:20.600092 7095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:20.607443 7095 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:20.607544 7095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:20.607708 7095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.65
I20240502 14:17:20.616312 7095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.58.65
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.65
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:20.617414 7095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:20.619009 7095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:20.629904 7101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:20.630429 7100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:20.630993 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:20.631237 7095 server_base.cc:1034] running on GCE node
I20240502 14:17:20.631762 7095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:20.632550 7095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:20.633741 7095 hybrid_clock.cc:648] HybridClock initialized: now 1714659440633690 us; error 44 us; skew 500 ppm
I20240502 14:17:20.634014 7095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:20.641566 7095 webserver.cc:460] Webserver started at http://127.21.58.65:33141/ using document root <none> and password file <none>
I20240502 14:17:20.642239 7095 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:20.642347 7095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:20.642637 7095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:17:20.645006 7095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f7ad4b3a72f04f259caff4473296ae72"
format_stamp: "Formatted at 2024-05-02 14:17:20 on dist-test-slave-wtgm"
I20240502 14:17:20.645697 7095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f7ad4b3a72f04f259caff4473296ae72"
format_stamp: "Formatted at 2024-05-02 14:17:20 on dist-test-slave-wtgm"
I20240502 14:17:20.650334 7095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:17:20.653378 7110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:20.654528 7095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:17:20.654685 7095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f7ad4b3a72f04f259caff4473296ae72"
format_stamp: "Formatted at 2024-05-02 14:17:20 on dist-test-slave-wtgm"
I20240502 14:17:20.654877 7095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:20.669317 7095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:20.670008 7095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:20.670253 7095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:20.671147 7095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:20.672492 7095 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:20.672587 7095 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:20.672706 7095 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:20.672780 7095 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:20.703094 7095 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.65:37021
I20240502 14:17:20.703193 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.65:37021 every 8 connection(s)
I20240502 14:17:20.704267 7095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:17:20.709928 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 7095
I20240502 14:17:20.710135 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:17:20.713878 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.66:0
--local_ip_for_outbound_sockets=127.21.58.66
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:17:20.717531 7224 heartbeater.cc:347] Connected to a master server at 127.21.58.126:39077
I20240502 14:17:20.717829 7224 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:20.718812 7224 heartbeater.cc:511] Master 127.21.58.126:39077 requested a full tablet report, sending...
I20240502 14:17:20.764397 7035 ts_manager.cc:194] Registered new tserver with Master: f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:20.766773 7035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.65:60489
W20240502 14:17:21.042430 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:21.042697 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:21.042791 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:21.049857 7228 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:21.049954 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:21.050074 7228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.66
I20240502 14:17:21.058457 7228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.58.66
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.66
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:21.059675 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:21.061295 7228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:21.071883 7234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:21.072340 7237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:21.072907 7235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:21.073184 7228 server_base.cc:1034] running on GCE node
I20240502 14:17:21.073705 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:21.074476 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:21.075671 7228 hybrid_clock.cc:648] HybridClock initialized: now 1714659441075623 us; error 36 us; skew 500 ppm
I20240502 14:17:21.075945 7228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:21.077980 7228 webserver.cc:460] Webserver started at http://127.21.58.66:42359/ using document root <none> and password file <none>
I20240502 14:17:21.078619 7228 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:21.078737 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:21.079027 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:17:21.081352 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e230e4b943ef49059c6a55ad1e667a02"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:21.082104 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e230e4b943ef49059c6a55ad1e667a02"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:21.086658 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 14:17:21.089751 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:21.090842 7228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:21.090989 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e230e4b943ef49059c6a55ad1e667a02"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:21.091261 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:21.133859 7228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:21.134624 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:21.134877 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:21.135807 7228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:21.137148 7228 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:21.137244 7228 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:21.137360 7228 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:21.137440 7228 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:21.170514 7228 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.66:36565
I20240502 14:17:21.170570 7356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.66:36565 every 8 connection(s)
I20240502 14:17:21.171839 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:17:21.175074 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 7228
I20240502 14:17:21.175336 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:17:21.178423 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.67:0
--local_ip_for_outbound_sockets=127.21.58.67
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:17:21.184520 7357 heartbeater.cc:347] Connected to a master server at 127.21.58.126:39077
I20240502 14:17:21.184789 7357 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:21.185513 7357 heartbeater.cc:511] Master 127.21.58.126:39077 requested a full tablet report, sending...
I20240502 14:17:21.245774 7035 ts_manager.cc:194] Registered new tserver with Master: e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565)
I20240502 14:17:21.246675 7035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.66:53333
W20240502 14:17:21.519270 7360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:21.519539 7360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:21.519685 7360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:21.526729 7360 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:21.526855 7360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:21.527020 7360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.67
I20240502 14:17:21.535640 7360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.58.67
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.67
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:21.536967 7360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:21.538599 7360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:21.550091 7367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:21.550134 7366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:21.550729 7369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:21.551446 7360 server_base.cc:1034] running on GCE node
I20240502 14:17:21.551980 7360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:21.552799 7360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:21.554004 7360 hybrid_clock.cc:648] HybridClock initialized: now 1714659441553963 us; error 45 us; skew 500 ppm
I20240502 14:17:21.554278 7360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:21.556298 7360 webserver.cc:460] Webserver started at http://127.21.58.67:41925/ using document root <none> and password file <none>
I20240502 14:17:21.556962 7360 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:21.557070 7360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:21.557355 7360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:17:21.559829 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "699831d1e03d4a109eb19f69687deac2"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:21.560541 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "699831d1e03d4a109eb19f69687deac2"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:21.565219 7360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:17:21.568620 7375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:21.569811 7360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:17:21.569962 7360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "699831d1e03d4a109eb19f69687deac2"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:21.570138 7360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:21.585013 7360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:21.585700 7360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:21.585943 7360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:21.586769 7360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:21.588122 7360 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:21.588218 7360 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:21.588325 7360 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:21.588394 7360 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:21.618877 7360 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.67:40049
I20240502 14:17:21.618927 7489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.67:40049 every 8 connection(s)
I20240502 14:17:21.620226 7360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:17:21.630281 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 7360
I20240502 14:17:21.630455 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:17:21.631925 7490 heartbeater.cc:347] Connected to a master server at 127.21.58.126:39077
I20240502 14:17:21.632194 7490 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:21.632941 7490 heartbeater.cc:511] Master 127.21.58.126:39077 requested a full tablet report, sending...
I20240502 14:17:21.633960 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:0
--local_ip_for_outbound_sockets=127.21.58.68
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:17:21.678323 7035 ts_manager.cc:194] Registered new tserver with Master: 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:21.679620 7035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.67:35761
I20240502 14:17:21.769948 7224 heartbeater.cc:503] Master 127.21.58.126:39077 was elected leader, sending a full tablet report...
W20240502 14:17:21.959453 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:21.959699 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:21.959852 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:21.966856 7494 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:21.966964 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:21.967103 7494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:17:21.975549 7494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:21.976652 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:21.978188 7494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:21.989024 7500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:21.989470 7499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:21.990053 7502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:21.990214 7494 server_base.cc:1034] running on GCE node
I20240502 14:17:21.990736 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:21.991549 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:21.992739 7494 hybrid_clock.cc:648] HybridClock initialized: now 1714659441992681 us; error 41 us; skew 500 ppm
I20240502 14:17:21.993014 7494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:21.994948 7494 webserver.cc:460] Webserver started at http://127.21.58.68:36897/ using document root <none> and password file <none>
I20240502 14:17:21.995638 7494 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:21.995775 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:21.996098 7494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:17:21.998416 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1b8d0ccec3714c3694d8285fcceb4448"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:21.999264 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1b8d0ccec3714c3694d8285fcceb4448"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:22.003784 7494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:17:22.006837 7508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:22.007997 7494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:17:22.008134 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1b8d0ccec3714c3694d8285fcceb4448"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:22.008318 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:22.029333 7494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:22.030031 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:22.030272 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:22.031147 7494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:22.032474 7494 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:22.032562 7494 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:22.032665 7494 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:22.032763 7494 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:22.062853 7494 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:38889
I20240502 14:17:22.062909 7622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:38889 every 8 connection(s)
I20240502 14:17:22.064108 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:17:22.073710 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 7494
I20240502 14:17:22.073916 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:17:22.076108 7623 heartbeater.cc:347] Connected to a master server at 127.21.58.126:39077
I20240502 14:17:22.076373 7623 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:22.077091 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:0
--local_ip_for_outbound_sockets=127.21.58.69
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:17:22.077119 7623 heartbeater.cc:511] Master 127.21.58.126:39077 requested a full tablet report, sending...
I20240502 14:17:22.132745 7035 ts_manager.cc:194] Registered new tserver with Master: 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889)
I20240502 14:17:22.133651 7035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:48675
I20240502 14:17:22.249780 7357 heartbeater.cc:503] Master 127.21.58.126:39077 was elected leader, sending a full tablet report...
W20240502 14:17:22.414819 7627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:22.415098 7627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:22.415241 7627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:22.422071 7627 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:22.422192 7627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:22.422333 7627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:17:22.430655 7627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=0
--tserver_master_addrs=127.21.58.126:39077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:22.431782 7627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:22.433334 7627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:22.444429 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:22.444808 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:22.445406 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:22.445673 7627 server_base.cc:1034] running on GCE node
I20240502 14:17:22.446192 7627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:22.446975 7627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:22.448170 7627 hybrid_clock.cc:648] HybridClock initialized: now 1714659442448146 us; error 48 us; skew 500 ppm
I20240502 14:17:22.448453 7627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:22.450430 7627 webserver.cc:460] Webserver started at http://127.21.58.69:45063/ using document root <none> and password file <none>
I20240502 14:17:22.451066 7627 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:22.451215 7627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:22.451507 7627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:17:22.453867 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "952df5ac1283460489e81cdb345007b7"
format_stamp: "Formatted at 2024-05-02 14:17:22 on dist-test-slave-wtgm"
I20240502 14:17:22.454587 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "952df5ac1283460489e81cdb345007b7"
format_stamp: "Formatted at 2024-05-02 14:17:22 on dist-test-slave-wtgm"
I20240502 14:17:22.460623 7627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:17:22.464236 7641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:22.465340 7627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:17:22.465478 7627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "952df5ac1283460489e81cdb345007b7"
format_stamp: "Formatted at 2024-05-02 14:17:22 on dist-test-slave-wtgm"
I20240502 14:17:22.465651 7627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:22.488286 7627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:22.488962 7627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:22.489205 7627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:22.490052 7627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:22.491436 7627 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:22.491533 7627 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:22.491638 7627 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:22.491709 7627 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:22.521612 7627 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:34127
I20240502 14:17:22.521660 7755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:34127 every 8 connection(s)
I20240502 14:17:22.522781 7627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:17:22.529726 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 7627
I20240502 14:17:22.529863 21737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:17:22.534372 7756 heartbeater.cc:347] Connected to a master server at 127.21.58.126:39077
I20240502 14:17:22.534602 7756 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:22.535301 7756 heartbeater.cc:511] Master 127.21.58.126:39077 requested a full tablet report, sending...
I20240502 14:17:22.577215 7035 ts_manager.cc:194] Registered new tserver with Master: 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127)
I20240502 14:17:22.578083 7035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:58393
I20240502 14:17:22.581825 21737 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:17:22.597575 21737 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:17:22.597718 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 7627
I20240502 14:17:22.605031 7751 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:22.681831 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 7627
I20240502 14:17:22.682490 7490 heartbeater.cc:503] Master 127.21.58.126:39077 was elected leader, sending a full tablet report...
I20240502 14:17:22.699502 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 7494
I20240502 14:17:22.707551 7618 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:22.785490 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 7494
I20240502 14:17:26.555951 7035 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50584:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 14:17:26.562634 7035 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:17:26.592976 7158 tablet_service.cc:1467] Processing CreateTablet for tablet 41104faa3b9a403e8e2316f8c40d4036 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.594345 7157 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a5a87d95a4657a2e8756962939ba2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.593570 7156 tablet_service.cc:1467] Processing CreateTablet for tablet c2c230f5dbc14ee886256e6710c817b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.595571 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 41104faa3b9a403e8e2316f8c40d4036 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.596923 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41104faa3b9a403e8e2316f8c40d4036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.599363 7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5a87d95a4657a2e8756962939ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.599823 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c230f5dbc14ee886256e6710c817b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.600055 7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41104faa3b9a403e8e2316f8c40d4036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.601130 7422 tablet_service.cc:1467] Processing CreateTablet for tablet c2c230f5dbc14ee886256e6710c817b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.601541 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c230f5dbc14ee886256e6710c817b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.601329 7290 tablet_service.cc:1467] Processing CreateTablet for tablet c2c230f5dbc14ee886256e6710c817b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.601966 7423 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a5a87d95a4657a2e8756962939ba2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.602325 7423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5a87d95a4657a2e8756962939ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.602684 7290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c230f5dbc14ee886256e6710c817b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.603503 7291 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a5a87d95a4657a2e8756962939ba2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.603989 7291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5a87d95a4657a2e8756962939ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.611295 7292 tablet_service.cc:1467] Processing CreateTablet for tablet 41104faa3b9a403e8e2316f8c40d4036 (DEFAULT_TABLE table=rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:26.611719 7292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41104faa3b9a403e8e2316f8c40d4036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:26.618235 7781 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:26.622018 7781 tablet_bootstrap.cc:654] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.623508 7781 log.cc:824] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:26.636715 7781 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:26.637033 7781 ts_tablet_manager.cc:1367] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20240502 14:17:26.639351 7780 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:26.640193 7783 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:26.642501 7783 tablet_bootstrap.cc:654] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.642637 7780 tablet_bootstrap.cc:654] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.643517 7783 log.cc:824] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:26.644917 7780 log.cc:824] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:26.646659 7781 raft_consensus.cc:348] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.646996 7781 raft_consensus.cc:374] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.647135 7781 raft_consensus.cc:729] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.647657 7780 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:26.647660 7783 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:26.647907 7780 ts_tablet_manager.cc:1367] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 14:17:26.647908 7783 ts_tablet_manager.cc:1367] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:17:26.647926 7781 consensus_queue.cc:260] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.649567 7781 ts_tablet_manager.cc:1398] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240502 14:17:26.649911 7781 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:26.651386 7781 tablet_bootstrap.cc:654] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.656759 7780 raft_consensus.cc:348] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.657150 7780 raft_consensus.cc:374] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.657236 7780 raft_consensus.cc:729] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.657931 7780 consensus_queue.cc:260] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.658428 7781 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:26.658582 7781 ts_tablet_manager.cc:1367] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 14:17:26.658311 7783 raft_consensus.cc:348] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.658716 7783 raft_consensus.cc:374] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.658820 7783 raft_consensus.cc:729] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.659538 7781 raft_consensus.cc:348] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.659513 7783 consensus_queue.cc:260] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.659724 7781 raft_consensus.cc:374] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.659786 7780 ts_tablet_manager.cc:1398] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20240502 14:17:26.659808 7781 raft_consensus.cc:729] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.660080 7781 consensus_queue.cc:260] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.660231 7780 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:26.660915 7781 ts_tablet_manager.cc:1398] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 14:17:26.661151 7781 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:26.662305 7780 tablet_bootstrap.cc:654] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.665231 7781 tablet_bootstrap.cc:654] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.667208 7783 ts_tablet_manager.cc:1398] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
I20240502 14:17:26.667641 7783 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:26.667992 7780 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:26.668116 7780 ts_tablet_manager.cc:1367] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:17:26.668993 7780 raft_consensus.cc:348] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.669206 7783 tablet_bootstrap.cc:654] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.669193 7780 raft_consensus.cc:374] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.669279 7780 raft_consensus.cc:729] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.669562 7780 consensus_queue.cc:260] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.670063 7780 ts_tablet_manager.cc:1398] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:26.670387 7780 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:26.672487 7780 tablet_bootstrap.cc:654] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.673794 7781 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:26.673954 7781 ts_tablet_manager.cc:1367] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240502 14:17:26.675693 7781 raft_consensus.cc:348] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.675874 7781 raft_consensus.cc:374] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.675961 7781 raft_consensus.cc:729] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.676182 7781 consensus_queue.cc:260] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.676692 7781 ts_tablet_manager.cc:1398] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20240502 14:17:26.676874 7358 tablet.cc:2369] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:17:26.678963 7780 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:26.679139 7780 ts_tablet_manager.cc:1367] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:17:26.679543 7783 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:26.679684 7783 ts_tablet_manager.cc:1367] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
I20240502 14:17:26.679811 7780 raft_consensus.cc:348] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.680035 7780 raft_consensus.cc:374] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.680110 7780 raft_consensus.cc:729] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.680330 7780 consensus_queue.cc:260] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.680621 7783 raft_consensus.cc:348] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.680819 7783 raft_consensus.cc:374] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.680878 7780 ts_tablet_manager.cc:1398] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:17:26.680915 7783 raft_consensus.cc:729] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.681121 7783 consensus_queue.cc:260] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.681520 7783 ts_tablet_manager.cc:1398] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:17:26.681749 7783 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:26.683177 7783 tablet_bootstrap.cc:654] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:26.684835 7783 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:26.684989 7783 ts_tablet_manager.cc:1367] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:17:26.685542 7783 raft_consensus.cc:348] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.685724 7783 raft_consensus.cc:374] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:26.685797 7783 raft_consensus.cc:729] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:26.686000 7783 consensus_queue.cc:260] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.686476 7783 ts_tablet_manager.cc:1398] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:17:26.702630 7788 raft_consensus.cc:482] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:26.702891 7788 raft_consensus.cc:504] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.704937 7788 leader_election.cc:288] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
W20240502 14:17:26.711370 7225 tablet.cc:2369] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:17:26.714205 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5a87d95a4657a2e8756962939ba2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2" is_pre_election: true
I20240502 14:17:26.714497 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5a87d95a4657a2e8756962939ba2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:26.714677 7444 raft_consensus.cc:2454] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:26.715090 7312 raft_consensus.cc:2454] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:26.715386 7113 leader_election.cc:302] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:26.715770 7788 raft_consensus.cc:2781] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:26.715914 7788 raft_consensus.cc:482] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:26.716030 7788 raft_consensus.cc:3037] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.718101 7788 raft_consensus.cc:504] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.718631 7788 leader_election.cc:288] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:26.719027 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5a87d95a4657a2e8756962939ba2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:26.719098 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5a87d95a4657a2e8756962939ba2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2"
I20240502 14:17:26.719228 7312 raft_consensus.cc:3037] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.719277 7444 raft_consensus.cc:3037] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.721297 7312 raft_consensus.cc:2454] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:26.721330 7444 raft_consensus.cc:2454] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:26.721643 7113 leader_election.cc:302] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:26.721894 7788 raft_consensus.cc:2781] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:26.722326 7788 raft_consensus.cc:686] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Becoming Leader. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Running, Role: LEADER
I20240502 14:17:26.722860 7788 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.727140 7034 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:26.774998 7789 raft_consensus.cc:482] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:26.775171 7789 raft_consensus.cc:504] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.776912 7789 leader_election.cc:288] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:26.783378 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41104faa3b9a403e8e2316f8c40d4036" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ad4b3a72f04f259caff4473296ae72" is_pre_election: true
I20240502 14:17:26.783476 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41104faa3b9a403e8e2316f8c40d4036" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:26.783699 7312 raft_consensus.cc:2454] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 0.
I20240502 14:17:26.783726 7179 raft_consensus.cc:2454] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 0.
I20240502 14:17:26.784169 7378 leader_election.cc:302] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, e230e4b943ef49059c6a55ad1e667a02; no voters:
I20240502 14:17:26.784497 7789 raft_consensus.cc:2781] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:26.784646 7789 raft_consensus.cc:482] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:26.784736 7789 raft_consensus.cc:3037] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.786650 7789 raft_consensus.cc:504] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.787218 7789 leader_election.cc:288] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:26.787552 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41104faa3b9a403e8e2316f8c40d4036" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:26.787645 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41104faa3b9a403e8e2316f8c40d4036" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
I20240502 14:17:26.787746 7312 raft_consensus.cc:3037] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.787812 7179 raft_consensus.cc:3037] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.789558 7312 raft_consensus.cc:2454] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 1.
I20240502 14:17:26.789685 7179 raft_consensus.cc:2454] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 1.
I20240502 14:17:26.789923 7378 leader_election.cc:302] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, e230e4b943ef49059c6a55ad1e667a02; no voters:
I20240502 14:17:26.790189 7789 raft_consensus.cc:2781] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:26.790563 7789 raft_consensus.cc:686] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Becoming Leader. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Running, Role: LEADER
I20240502 14:17:26.790977 7789 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } }
I20240502 14:17:26.795495 7034 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 699831d1e03d4a109eb19f69687deac2 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: UNKNOWN } } }
W20240502 14:17:26.875255 7491 tablet.cc:2369] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:17:26.905411 7788 raft_consensus.cc:482] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:26.905673 7788 raft_consensus.cc:504] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.906472 7788 leader_election.cc:288] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:26.907181 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c230f5dbc14ee886256e6710c817b9" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:26.907233 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c230f5dbc14ee886256e6710c817b9" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2" is_pre_election: true
I20240502 14:17:26.907459 7312 raft_consensus.cc:2454] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:26.907538 7444 raft_consensus.cc:2454] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:26.907980 7113 leader_election.cc:302] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:26.908433 7788 raft_consensus.cc:2781] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:26.908530 7788 raft_consensus.cc:482] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:26.908634 7788 raft_consensus.cc:3037] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.911123 7788 raft_consensus.cc:504] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.911626 7788 leader_election.cc:288] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:26.912050 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c230f5dbc14ee886256e6710c817b9" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:26.912078 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c230f5dbc14ee886256e6710c817b9" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2"
I20240502 14:17:26.912212 7312 raft_consensus.cc:3037] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.912287 7444 raft_consensus.cc:3037] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:26.914130 7444 raft_consensus.cc:2454] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:26.914252 7312 raft_consensus.cc:2454] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:26.914499 7113 leader_election.cc:302] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:26.914788 7788 raft_consensus.cc:2781] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:26.914952 7788 raft_consensus.cc:686] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Becoming Leader. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Running, Role: LEADER
I20240502 14:17:26.915244 7788 consensus_queue.cc:237] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:26.917661 7034 catalog_manager.cc:5484] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:27.155913 7788 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:27.164783 7788 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:27.313167 7797 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:27.318635 7797 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:27.337419 7789 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:27.343945 7789 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:17:27.344360 7805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:27.344708 7805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:27.353199 7805 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 259.297 ms
time per row: 0.129648 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:17:28.676185 7035 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50584:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 14:17:28.676935 7035 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:17:28.685036 7156 tablet_service.cc:1467] Processing CreateTablet for tablet c4830003d84f487d80b7744ca03c1fbe (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.685531 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4830003d84f487d80b7744ca03c1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.685668 7157 tablet_service.cc:1467] Processing CreateTablet for tablet 67c65d90099b47f0859ce598e7e727c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.685775 7292 tablet_service.cc:1467] Processing CreateTablet for tablet c4830003d84f487d80b7744ca03c1fbe (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.686033 7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c65d90099b47f0859ce598e7e727c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.686226 7292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4830003d84f487d80b7744ca03c1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.686362 7422 tablet_service.cc:1467] Processing CreateTablet for tablet c4830003d84f487d80b7744ca03c1fbe (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.686761 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4830003d84f487d80b7744ca03c1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.688158 7158 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccb2c50626e4d3790f1cc006a4a7446 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.688496 7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccb2c50626e4d3790f1cc006a4a7446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.691911 7291 tablet_service.cc:1467] Processing CreateTablet for tablet 67c65d90099b47f0859ce598e7e727c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.692272 7291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c65d90099b47f0859ce598e7e727c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.692318 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 67c65d90099b47f0859ce598e7e727c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.692798 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c65d90099b47f0859ce598e7e727c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.695506 7290 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccb2c50626e4d3790f1cc006a4a7446 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.695552 7423 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccb2c50626e4d3790f1cc006a4a7446 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:28.695909 7290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccb2c50626e4d3790f1cc006a4a7446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.696010 7423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccb2c50626e4d3790f1cc006a4a7446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:28.708530 7873 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:28.711560 7873 tablet_bootstrap.cc:654] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.715440 7873 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:28.715586 7873 ts_tablet_manager.cc:1367] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:17:28.716311 7873 raft_consensus.cc:348] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.716516 7873 raft_consensus.cc:374] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.717515 7871 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:28.718302 7873 raft_consensus.cc:729] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.718580 7873 consensus_queue.cc:260] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.719463 7871 tablet_bootstrap.cc:654] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.720002 7873 ts_tablet_manager.cc:1398] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:17:28.720269 7873 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:28.720700 7872 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:28.721808 7873 tablet_bootstrap.cc:654] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.722332 7872 tablet_bootstrap.cc:654] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.724172 7871 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:28.724186 7873 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:28.724350 7873 ts_tablet_manager.cc:1367] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:17:28.724388 7871 ts_tablet_manager.cc:1367] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:17:28.724985 7873 raft_consensus.cc:348] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.725162 7872 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:28.725162 7873 raft_consensus.cc:374] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.725106 7871 raft_consensus.cc:348] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.725235 7873 raft_consensus.cc:729] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.725317 7871 raft_consensus.cc:374] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.725339 7872 ts_tablet_manager.cc:1367] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:17:28.725476 7871 raft_consensus.cc:729] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.725474 7873 consensus_queue.cc:260] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.725746 7871 consensus_queue.cc:260] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.726018 7873 ts_tablet_manager.cc:1398] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:17:28.726078 7872 raft_consensus.cc:348] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.726310 7873 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:28.726301 7872 raft_consensus.cc:374] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.726395 7872 raft_consensus.cc:729] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.726534 7871 ts_tablet_manager.cc:1398] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:28.726650 7872 consensus_queue.cc:260] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.727216 7872 ts_tablet_manager.cc:1398] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:17:28.727519 7872 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:28.727810 7873 tablet_bootstrap.cc:654] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.729059 7872 tablet_bootstrap.cc:654] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.733682 7871 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:28.735832 7871 tablet_bootstrap.cc:654] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.736063 7873 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:28.736192 7873 ts_tablet_manager.cc:1367] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:17:28.736734 7873 raft_consensus.cc:348] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.736935 7873 raft_consensus.cc:374] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.737082 7873 raft_consensus.cc:729] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.737352 7873 consensus_queue.cc:260] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.737887 7871 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:28.737969 7873 ts_tablet_manager.cc:1398] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:28.738046 7871 ts_tablet_manager.cc:1367] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 14:17:28.738636 7872 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:28.738700 7871 raft_consensus.cc:348] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.738935 7871 raft_consensus.cc:374] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.739014 7871 raft_consensus.cc:729] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.739249 7871 consensus_queue.cc:260] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.739800 7871 ts_tablet_manager.cc:1398] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:28.740042 7871 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:28.740134 7872 ts_tablet_manager.cc:1367] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240502 14:17:28.741925 7871 tablet_bootstrap.cc:654] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.742313 7872 raft_consensus.cc:348] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.742496 7872 raft_consensus.cc:374] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.742573 7872 raft_consensus.cc:729] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.742802 7872 consensus_queue.cc:260] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.743429 7872 ts_tablet_manager.cc:1398] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:17:28.743691 7872 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:28.745123 7872 tablet_bootstrap.cc:654] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:28.748661 7872 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:28.748796 7872 ts_tablet_manager.cc:1367] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:17:28.749315 7872 raft_consensus.cc:348] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.749457 7872 raft_consensus.cc:374] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.749549 7872 raft_consensus.cc:729] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.749760 7872 consensus_queue.cc:260] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.750224 7872 ts_tablet_manager.cc:1398] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:17:28.755229 7871 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:28.755358 7871 ts_tablet_manager.cc:1367] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.008s
I20240502 14:17:28.755913 7871 raft_consensus.cc:348] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.756059 7871 raft_consensus.cc:374] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:28.756153 7871 raft_consensus.cc:729] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:28.756341 7871 consensus_queue.cc:260] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.756721 7871 ts_tablet_manager.cc:1398] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:17:28.759927 7854 raft_consensus.cc:482] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:28.760039 7854 raft_consensus.cc:504] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.760571 7854 leader_election.cc:288] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:28.760999 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccb2c50626e4d3790f1cc006a4a7446" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:28.761037 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccb2c50626e4d3790f1cc006a4a7446" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2" is_pre_election: true
I20240502 14:17:28.761220 7443 raft_consensus.cc:2454] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:28.761221 7311 raft_consensus.cc:2454] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:28.761616 7113 leader_election.cc:302] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:28.761852 7854 raft_consensus.cc:2781] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:28.761973 7854 raft_consensus.cc:482] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:28.762065 7854 raft_consensus.cc:3037] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.764091 7854 raft_consensus.cc:504] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.764598 7854 leader_election.cc:288] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:28.764911 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccb2c50626e4d3790f1cc006a4a7446" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:28.765017 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccb2c50626e4d3790f1cc006a4a7446" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2"
I20240502 14:17:28.765090 7311 raft_consensus.cc:3037] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.765218 7443 raft_consensus.cc:3037] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.766930 7311 raft_consensus.cc:2454] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:28.767015 7443 raft_consensus.cc:2454] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:28.767303 7113 leader_election.cc:302] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:28.767571 7854 raft_consensus.cc:2781] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:28.767722 7854 raft_consensus.cc:686] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Becoming Leader. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Running, Role: LEADER
I20240502 14:17:28.767977 7854 consensus_queue.cc:237] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.770342 7033 catalog_manager.cc:5484] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:28.859817 7854 raft_consensus.cc:482] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:28.860065 7854 raft_consensus.cc:504] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.860767 7854 leader_election.cc:288] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:28.861248 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:28.861335 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2" is_pre_election: true
I20240502 14:17:28.861483 7311 raft_consensus.cc:2454] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:28.861594 7443 raft_consensus.cc:2454] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:28.861879 7113 leader_election.cc:302] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:28.862157 7854 raft_consensus.cc:2781] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:28.862262 7854 raft_consensus.cc:482] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:28.862365 7854 raft_consensus.cc:3037] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.864230 7854 raft_consensus.cc:504] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.864768 7854 leader_election.cc:288] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:28.865103 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:28.865175 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2"
I20240502 14:17:28.865280 7311 raft_consensus.cc:3037] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.865378 7443 raft_consensus.cc:3037] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.867106 7311 raft_consensus.cc:2454] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:28.867237 7443 raft_consensus.cc:2454] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:28.867494 7113 leader_election.cc:302] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:28.867800 7854 raft_consensus.cc:2781] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:28.867952 7854 raft_consensus.cc:686] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Becoming Leader. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Running, Role: LEADER
I20240502 14:17:28.868211 7854 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.870729 7033 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:28.904523 7854 raft_consensus.cc:482] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:28.904747 7854 raft_consensus.cc:504] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.905501 7854 leader_election.cc:288] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:28.906085 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4830003d84f487d80b7744ca03c1fbe" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:28.906323 7311 raft_consensus.cc:2454] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:28.906360 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4830003d84f487d80b7744ca03c1fbe" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2" is_pre_election: true
I20240502 14:17:28.906600 7443 raft_consensus.cc:2454] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:28.907043 7113 leader_election.cc:302] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:28.907368 7854 raft_consensus.cc:2781] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:28.907526 7854 raft_consensus.cc:482] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:28.907641 7854 raft_consensus.cc:3037] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.909584 7854 raft_consensus.cc:504] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.910130 7854 leader_election.cc:288] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:28.910535 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4830003d84f487d80b7744ca03c1fbe" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:28.910610 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4830003d84f487d80b7744ca03c1fbe" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2"
I20240502 14:17:28.910708 7311 raft_consensus.cc:3037] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.910815 7443 raft_consensus.cc:3037] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:28.912570 7311 raft_consensus.cc:2454] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:28.912655 7443 raft_consensus.cc:2454] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:28.912937 7113 leader_election.cc:302] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:28.913221 7854 raft_consensus.cc:2781] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:28.913349 7854 raft_consensus.cc:686] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Becoming Leader. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Running, Role: LEADER
I20240502 14:17:28.913609 7854 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:28.916081 7033 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:29.172304 7854 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:29.194420 7883 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:17:29.288432 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:29.288791 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:29.296520 7889 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:17:29.348299 7895 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:29.359163 7854 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:29.433171 7443 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:17:29.434355 7854 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:29.436808 7312 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:17:29.438040 7883 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:29.486141 7892 mvcc.cc:205] Tried to move back new op lower bound from 7023245104858451968 to 7023245102752055296. Current Snapshot: MvccSnapshot[applied={T|T < 7023245104858451968}]
INSERT report
rows total: 2000
time total: 209.789 ms
time per row: 0.104895 ms
I20240502 14:17:29.648372 7035 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50584:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 14:17:29.649152 7035 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:17:29.657227 7290 tablet_service.cc:1467] Processing CreateTablet for tablet d0cd8c0d16f844edb70cec8092dd7f05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.657411 7156 tablet_service.cc:1467] Processing CreateTablet for tablet d0cd8c0d16f844edb70cec8092dd7f05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.657774 7290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cd8c0d16f844edb70cec8092dd7f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.657860 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cd8c0d16f844edb70cec8092dd7f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.658473 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 6af3a6191cd34b3dac8900eb0ef265e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.658628 7422 tablet_service.cc:1467] Processing CreateTablet for tablet d0cd8c0d16f844edb70cec8092dd7f05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.658910 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af3a6191cd34b3dac8900eb0ef265e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.660120 7157 tablet_service.cc:1467] Processing CreateTablet for tablet 6af3a6191cd34b3dac8900eb0ef265e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.660468 7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af3a6191cd34b3dac8900eb0ef265e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.663514 7423 tablet_service.cc:1467] Processing CreateTablet for tablet 3293190336514711801388ea1e1a1583 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.663898 7423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3293190336514711801388ea1e1a1583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.667579 7158 tablet_service.cc:1467] Processing CreateTablet for tablet 3293190336514711801388ea1e1a1583 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.668102 7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3293190336514711801388ea1e1a1583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.669597 7948 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:29.671453 7948 tablet_bootstrap.cc:654] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.671530 7291 tablet_service.cc:1467] Processing CreateTablet for tablet 6af3a6191cd34b3dac8900eb0ef265e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.671954 7291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af3a6191cd34b3dac8900eb0ef265e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.673640 7948 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:29.673772 7948 ts_tablet_manager.cc:1367] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:17:29.674491 7292 tablet_service.cc:1467] Processing CreateTablet for tablet 3293190336514711801388ea1e1a1583 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:17:29.674517 7948 raft_consensus.cc:348] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.674705 7948 raft_consensus.cc:374] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.674793 7948 raft_consensus.cc:729] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.674870 7292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3293190336514711801388ea1e1a1583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.675033 7948 consensus_queue.cc:260] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.675657 7948 ts_tablet_manager.cc:1398] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:29.679566 7949 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:29.680802 7951 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:29.680970 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cd8c0d16f844edb70cec8092dd7f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:29.681617 7949 tablet_bootstrap.cc:654] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.683403 7951 tablet_bootstrap.cc:654] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.683859 7949 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:29.684006 7949 ts_tablet_manager.cc:1367] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:17:29.684888 7949 raft_consensus.cc:348] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.685123 7949 raft_consensus.cc:374] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.685217 7949 raft_consensus.cc:729] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.685534 7949 consensus_queue.cc:260] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.686103 7949 ts_tablet_manager.cc:1398] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:29.686640 7949 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:29.688211 7949 tablet_bootstrap.cc:654] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.689054 7951 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:29.689200 7951 ts_tablet_manager.cc:1367] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:17:29.690963 7951 raft_consensus.cc:348] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.691236 7951 raft_consensus.cc:374] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.691406 7951 raft_consensus.cc:729] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.691675 7951 consensus_queue.cc:260] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.692363 7951 ts_tablet_manager.cc:1398] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:17:29.692718 7951 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:29.693620 7949 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:29.693787 7949 ts_tablet_manager.cc:1367] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 14:17:29.694348 7951 tablet_bootstrap.cc:654] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.694517 7949 raft_consensus.cc:348] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.694756 7949 raft_consensus.cc:374] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.694820 7949 raft_consensus.cc:729] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.695195 7948 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:29.695152 7949 consensus_queue.cc:260] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.696825 7948 tablet_bootstrap.cc:654] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.696934 7949 ts_tablet_manager.cc:1398] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:17:29.697322 7949 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: Bootstrap starting.
I20240502 14:17:29.699031 7949 tablet_bootstrap.cc:654] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.699162 7951 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:29.699322 7951 ts_tablet_manager.cc:1367] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:17:29.700268 7951 raft_consensus.cc:348] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.700585 7951 raft_consensus.cc:374] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.700681 7951 raft_consensus.cc:729] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.701035 7951 consensus_queue.cc:260] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.701799 7951 ts_tablet_manager.cc:1398] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:17:29.702198 7951 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: Bootstrap starting.
I20240502 14:17:29.704332 7951 tablet_bootstrap.cc:654] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.706405 7948 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:29.706605 7948 ts_tablet_manager.cc:1367] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 14:17:29.706939 7951 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: No bootstrap required, opened a new log
I20240502 14:17:29.707139 7951 ts_tablet_manager.cc:1367] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:17:29.707490 7948 raft_consensus.cc:348] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.707767 7948 raft_consensus.cc:374] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.707876 7948 raft_consensus.cc:729] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.708101 7951 raft_consensus.cc:348] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.708348 7951 raft_consensus.cc:374] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.708447 7951 raft_consensus.cc:729] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.708745 7951 consensus_queue.cc:260] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.709635 7951 ts_tablet_manager.cc:1398] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:29.710228 7948 consensus_queue.cc:260] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.710938 7949 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: No bootstrap required, opened a new log
I20240502 14:17:29.711117 7949 ts_tablet_manager.cc:1367] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20240502 14:17:29.711478 7948 ts_tablet_manager.cc:1398] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:17:29.711797 7948 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: Bootstrap starting.
I20240502 14:17:29.711853 7949 raft_consensus.cc:348] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.712039 7949 raft_consensus.cc:374] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.712122 7949 raft_consensus.cc:729] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.712332 7949 consensus_queue.cc:260] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.712594 7883 raft_consensus.cc:482] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:29.712743 7883 raft_consensus.cc:504] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.712874 7949 ts_tablet_manager.cc:1398] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:17:29.713531 7948 tablet_bootstrap.cc:654] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: Neither blocks nor log segments found. Creating new log.
I20240502 14:17:29.713495 7883 leader_election.cc:288] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:29.714449 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af3a6191cd34b3dac8900eb0ef265e2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:29.714674 7312 raft_consensus.cc:2454] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:29.715230 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af3a6191cd34b3dac8900eb0ef265e2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2" is_pre_election: true
I20240502 14:17:29.715425 7444 raft_consensus.cc:2454] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 0.
I20240502 14:17:29.716568 7113 leader_election.cc:302] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:29.718132 7883 raft_consensus.cc:2781] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:29.718246 7883 raft_consensus.cc:482] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:29.718322 7883 raft_consensus.cc:3037] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.721539 7883 raft_consensus.cc:504] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.722074 7948 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: No bootstrap required, opened a new log
I20240502 14:17:29.722223 7948 ts_tablet_manager.cc:1367] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240502 14:17:29.722533 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af3a6191cd34b3dac8900eb0ef265e2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:29.722720 7312 raft_consensus.cc:3037] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.723030 7948 raft_consensus.cc:348] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.723230 7948 raft_consensus.cc:374] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:17:29.723333 7948 raft_consensus.cc:729] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Initialized, Role: FOLLOWER
I20240502 14:17:29.723572 7948 consensus_queue.cc:260] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.724100 7948 ts_tablet_manager.cc:1398] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:29.725611 7312 raft_consensus.cc:2454] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:29.725997 7113 leader_election.cc:302] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:29.726275 7895 raft_consensus.cc:2781] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:29.726444 7895 raft_consensus.cc:686] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Becoming Leader. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Running, Role: LEADER
I20240502 14:17:29.726800 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af3a6191cd34b3dac8900eb0ef265e2" candidate_uuid: "f7ad4b3a72f04f259caff4473296ae72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2"
I20240502 14:17:29.726768 7895 consensus_queue.cc:237] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.727005 7444 raft_consensus.cc:3037] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.729774 7444 raft_consensus.cc:2454] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ad4b3a72f04f259caff4473296ae72 in term 1.
I20240502 14:17:29.730026 7883 leader_election.cc:288] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [CANDIDATE]: Term 1 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:29.730300 7817 raft_consensus.cc:482] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:29.730471 7817 raft_consensus.cc:504] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.731217 7817 leader_election.cc:288] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021), e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565)
I20240502 14:17:29.731846 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3293190336514711801388ea1e1a1583" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02" is_pre_election: true
I20240502 14:17:29.731964 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3293190336514711801388ea1e1a1583" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ad4b3a72f04f259caff4473296ae72" is_pre_election: true
I20240502 14:17:29.732120 7179 raft_consensus.cc:2454] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 0.
I20240502 14:17:29.732141 7312 raft_consensus.cc:2454] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 0.
I20240502 14:17:29.732527 7376 leader_election.cc:302] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:29.732791 7817 raft_consensus.cc:2781] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:29.732898 7817 raft_consensus.cc:482] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:29.732991 7817 raft_consensus.cc:3037] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.733476 7029 catalog_manager.cc:5484] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65). New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:29.735846 7817 raft_consensus.cc:504] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.736820 7817 leader_election.cc:288] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 election: Requested vote from peers f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021), e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565)
I20240502 14:17:29.736789 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3293190336514711801388ea1e1a1583" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
I20240502 14:17:29.736997 7179 raft_consensus.cc:3037] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.739424 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3293190336514711801388ea1e1a1583" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:29.739646 7312 raft_consensus.cc:3037] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.739845 7179 raft_consensus.cc:2454] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 1.
I20240502 14:17:29.740185 7376 leader_election.cc:302] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:29.740489 7817 raft_consensus.cc:2781] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:29.740981 7817 raft_consensus.cc:686] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Becoming Leader. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Running, Role: LEADER
I20240502 14:17:29.741340 7817 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.742280 7312 raft_consensus.cc:2454] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 1.
I20240502 14:17:29.744124 7029 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 699831d1e03d4a109eb19f69687deac2 (127.21.58.67). New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } }
I20240502 14:17:29.957239 7875 raft_consensus.cc:482] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:17:29.957461 7875 raft_consensus.cc:504] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.959142 7875 leader_election.cc:288] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:29.966388 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05" candidate_uuid: "e230e4b943ef49059c6a55ad1e667a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2" is_pre_election: true
I20240502 14:17:29.966439 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05" candidate_uuid: "e230e4b943ef49059c6a55ad1e667a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ad4b3a72f04f259caff4473296ae72" is_pre_election: true
I20240502 14:17:29.966611 7444 raft_consensus.cc:2454] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e230e4b943ef49059c6a55ad1e667a02 in term 0.
I20240502 14:17:29.966647 7179 raft_consensus.cc:2454] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e230e4b943ef49059c6a55ad1e667a02 in term 0.
I20240502 14:17:29.967114 7246 leader_election.cc:302] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, e230e4b943ef49059c6a55ad1e667a02; no voters:
I20240502 14:17:29.967491 7875 raft_consensus.cc:2781] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:17:29.967610 7875 raft_consensus.cc:482] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:17:29.967701 7875 raft_consensus.cc:3037] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.969633 7875 raft_consensus.cc:504] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.970239 7875 leader_election.cc:288] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [CANDIDATE]: Term 1 election: Requested vote from peers f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021), 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:29.970615 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05" candidate_uuid: "e230e4b943ef49059c6a55ad1e667a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
I20240502 14:17:29.970702 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05" candidate_uuid: "e230e4b943ef49059c6a55ad1e667a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699831d1e03d4a109eb19f69687deac2"
I20240502 14:17:29.970794 7179 raft_consensus.cc:3037] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.970906 7444 raft_consensus.cc:3037] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:17:29.972702 7179 raft_consensus.cc:2454] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e230e4b943ef49059c6a55ad1e667a02 in term 1.
I20240502 14:17:29.972798 7444 raft_consensus.cc:2454] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e230e4b943ef49059c6a55ad1e667a02 in term 1.
I20240502 14:17:29.973112 7244 leader_election.cc:302] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:29.973434 7875 raft_consensus.cc:2781] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:17:29.973838 7875 raft_consensus.cc:686] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Becoming Leader. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Running, Role: LEADER
I20240502 14:17:29.974272 7875 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:29.977509 7029 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66). New cstate: current_term: 1 leader_uuid: "e230e4b943ef49059c6a55ad1e667a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:30.143441 7883 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:30.150439 7883 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:30.174088 7817 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:30.179837 7961 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:17:30.385941 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:30.386355 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:30.399268 7968 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:17:30.468858 7875 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:17:30.477787 7875 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 205.43 ms
time per row: 0.102715 ms
I20240502 14:17:30.747814 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.69:34127
--local_ip_for_outbound_sockets=127.21.58.69
--tserver_master_addrs=127.21.58.126:39077
--webserver_port=45063
--webserver_interface=127.21.58.69
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:17:31.073347 8029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:31.073679 8029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:31.073817 8029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:31.081125 8029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:31.081254 8029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:31.081406 8029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.69
I20240502 14:17:31.090701 8029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.69:34127
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.58.69
--webserver_port=45063
--tserver_master_addrs=127.21.58.126:39077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.69
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:31.092051 8029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:31.093838 8029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:31.106323 8035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:31.106891 8034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:31.106367 8037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:31.111600 8029 server_base.cc:1034] running on GCE node
I20240502 14:17:31.112144 8029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:31.113039 8029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:31.114240 8029 hybrid_clock.cc:648] HybridClock initialized: now 1714659451114179 us; error 45 us; skew 500 ppm
I20240502 14:17:31.114564 8029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:31.116665 8029 webserver.cc:460] Webserver started at http://127.21.58.69:45063/ using document root <none> and password file <none>
I20240502 14:17:31.117322 8029 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:31.117434 8029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:31.122362 8029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:17:31.125003 8043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:31.126222 8029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:17:31.126370 8029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "952df5ac1283460489e81cdb345007b7"
format_stamp: "Formatted at 2024-05-02 14:17:22 on dist-test-slave-wtgm"
I20240502 14:17:31.127143 8029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:31.146528 8029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:31.147322 8029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:31.147617 8029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:31.148463 8029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:31.149879 8029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:31.149953 8029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:31.150079 8029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:31.150131 8029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:31.189203 8029 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.69:34127
I20240502 14:17:31.189265 8155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.69:34127 every 8 connection(s)
I20240502 14:17:31.190513 8029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:17:31.196347 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 8029
I20240502 14:17:31.197198 21737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cgK4A/build/asan/bin/kudu
/tmp/dist-test-task3cgK4A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.58.68:38889
--local_ip_for_outbound_sockets=127.21.58.68
--tserver_master_addrs=127.21.58.126:39077
--webserver_port=36897
--webserver_interface=127.21.58.68
--builtin_ntp_servers=127.21.58.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:17:31.205543 8156 heartbeater.cc:347] Connected to a master server at 127.21.58.126:39077
I20240502 14:17:31.205797 8156 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:31.206624 8156 heartbeater.cc:511] Master 127.21.58.126:39077 requested a full tablet report, sending...
I20240502 14:17:31.208365 7035 ts_manager.cc:194] Re-registered known tserver with Master: 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127)
I20240502 14:17:31.209326 7035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.69:37935
W20240502 14:17:31.527951 8160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:17:31.528198 8160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:17:31.528283 8160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:17:31.535691 8160 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:17:31.535811 8160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:17:31.535977 8160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.58.68
I20240502 14:17:31.545405 8160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.58.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.58.68:38889
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.58.68
--webserver_port=36897
--tserver_master_addrs=127.21.58.126:39077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.58.68
--log_dir=/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:17:31.546712 8160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:17:31.548373 8160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:17:31.559567 8165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:31.559600 8168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:17:31.560184 8166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:17:31.560859 8160 server_base.cc:1034] running on GCE node
I20240502 14:17:31.561358 8160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:17:31.562160 8160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:17:31.563355 8160 hybrid_clock.cc:648] HybridClock initialized: now 1714659451563319 us; error 26 us; skew 500 ppm
I20240502 14:17:31.563629 8160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:17:31.565601 8160 webserver.cc:460] Webserver started at http://127.21.58.68:36897/ using document root <none> and password file <none>
I20240502 14:17:31.566255 8160 fs_manager.cc:359] Metadata directory not provided
I20240502 14:17:31.566373 8160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:17:31.571630 8160 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:17:31.574124 8174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:31.575441 8160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:17:31.575577 8160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1b8d0ccec3714c3694d8285fcceb4448"
format_stamp: "Formatted at 2024-05-02 14:17:21 on dist-test-slave-wtgm"
I20240502 14:17:31.576344 8160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:17:31.604897 8160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:17:31.605674 8160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:17:31.605931 8160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:17:31.606761 8160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:17:31.608186 8160 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:17:31.608263 8160 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:31.608397 8160 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:17:31.608453 8160 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:17:31.638692 8160 rpc_server.cc:307] RPC server started. Bound to: 127.21.58.68:38889
I20240502 14:17:31.638754 8286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.58.68:38889 every 8 connection(s)
I20240502 14:17:31.640133 8160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:17:31.649839 21737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cgK4A/build/asan/bin/kudu as pid 8160
I20240502 14:17:31.651772 8287 heartbeater.cc:347] Connected to a master server at 127.21.58.126:39077
I20240502 14:17:31.652035 8287 heartbeater.cc:464] Registering TS with master...
I20240502 14:17:31.652837 8287 heartbeater.cc:511] Master 127.21.58.126:39077 requested a full tablet report, sending...
I20240502 14:17:31.654186 7030 ts_manager.cc:194] Re-registered known tserver with Master: 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889)
I20240502 14:17:31.654929 7030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.58.68:37553
I20240502 14:17:32.125595 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.131031 7158 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:17:32.134788 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:32.136695 7422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:17:32.143599 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.212234 8156 heartbeater.cc:503] Master 127.21.58.126:39077 was elected leader, sending a full tablet report...
I20240502 14:17:32.323946 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.325176 7422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:17:32.325596 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.329635 7158 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:17:32.340277 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:32.532567 7422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:17:32.532584 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.533200 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:32.535167 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.544651 7158 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:17:32.657155 8287 heartbeater.cc:503] Master 127.21.58.126:39077 was elected leader, sending a full tablet report...
I20240502 14:17:32.703022 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.703202 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:32.705247 7158 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:17:32.713265 7422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:17:32.722488 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:32.799397 7178 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:32.802488 7311 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.802837 7444 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.803445 8435 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.803861 7895 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.811966 8435 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
W20240502 14:17:32.812613 7113 consensus_peers.cc:485] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f9a5a87d95a4657a2e8756962939ba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:32.812680 7311 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.813432 7444 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.817713 7034 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.830564 7178 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:32.833081 7311 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.833626 8436 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.834035 7444 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.835466 8436 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.836372 8435 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.836966 7311 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
W20240502 14:17:32.840353 7113 consensus_peers.cc:485] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 -> Peer 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127): Couldn't send request to peer 952df5ac1283460489e81cdb345007b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67c65d90099b47f0859ce598e7e727c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:32.842022 7030 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.846980 7442 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.858371 7311 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:32.861055 7178 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.861800 8458 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.862568 7442 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.863205 8458 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.864889 7875 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.865375 7178 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.867504 7442 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.870391 7034 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "e230e4b943ef49059c6a55ad1e667a02" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
W20240502 14:17:32.880365 7246 consensus_peers.cc:485] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0cd8c0d16f844edb70cec8092dd7f05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:32.891819 7442 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:32.894129 7178 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.894851 7975 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.896138 7311 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.896632 8471 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.898115 7975 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.899644 7178 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
W20240502 14:17:32.903348 7378 consensus_peers.cc:485] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 -> Peer 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127): Couldn't send request to peer 952df5ac1283460489e81cdb345007b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3293190336514711801388ea1e1a1583. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:32.906385 7030 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:32.909363 7311 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.924701 7178 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:32.926787 7311 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.926846 7442 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:17:32.926978 7113 consensus_peers.cc:485] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4830003d84f487d80b7744ca03c1fbe. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:32.927320 7895 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.927608 8435 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.930133 7895 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.930717 7442 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.931349 7311 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:32.934950 7034 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index -1 to 4, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:32.947957 7442 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:32.949977 7311 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.950201 7178 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:32.950681 8475 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:17:32.950950 7378 consensus_peers.cc:485] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 -> Peer 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127): Couldn't send request to peer 952df5ac1283460489e81cdb345007b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41104faa3b9a403e8e2316f8c40d4036. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:32.950989 7975 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:32.953599 7975 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.954218 7178 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.954265 7311 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:32.959851 7026 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index -1 to 4, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:33.244776 8542 ts_tablet_manager.cc:902] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:33.246717 8542 tablet_copy_client.cc:308] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:33.254294 7199 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet 2f9a5a87d95a4657a2e8756962939ba2 from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:49317)
I20240502 14:17:33.254738 7199 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet 2f9a5a87d95a4657a2e8756962939ba2 from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:49317: session id = 1b8d0ccec3714c3694d8285fcceb4448-2f9a5a87d95a4657a2e8756962939ba2
I20240502 14:17:33.257838 7199 tablet_copy_source_session.cc:205] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:33.263209 8542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5a87d95a4657a2e8756962939ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:33.273370 8559 ts_tablet_manager.cc:902] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:33.275480 8559 tablet_copy_client.cc:308] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:33.275895 8542 tablet_copy_client.cc:783] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:33.276268 8542 tablet_copy_client.cc:647] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:33.278467 8542 tablet_copy_client.cc:523] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:33.282063 8542 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:33.282037 7199 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet 67c65d90099b47f0859ce598e7e727c6 from peer 952df5ac1283460489e81cdb345007b7 ({username='slave'} at 127.21.58.69:35189)
I20240502 14:17:33.282200 7199 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet 67c65d90099b47f0859ce598e7e727c6 from peer 952df5ac1283460489e81cdb345007b7 at {username='slave'} at 127.21.58.69:35189: session id = 952df5ac1283460489e81cdb345007b7-67c65d90099b47f0859ce598e7e727c6
I20240502 14:17:33.284106 7199 tablet_copy_source_session.cc:205] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:33.288669 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c65d90099b47f0859ce598e7e727c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:33.293882 8542 log.cc:824] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:33.302513 8559 tablet_copy_client.cc:783] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:33.302907 8559 tablet_copy_client.cc:647] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:33.304925 8559 tablet_copy_client.cc:523] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:33.309129 8559 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: Bootstrap starting.
I20240502 14:17:33.321683 8559 log.cc:824] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: Log is configured to *not* fsync() on all Append() calls
I20240502 14:17:33.339540 8542 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:33.340167 8542 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:33.340631 8542 ts_tablet_manager.cc:1367] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.059s user 0.051s sys 0.008s
I20240502 14:17:33.346824 8542 raft_consensus.cc:348] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:33.347338 8542 raft_consensus.cc:729] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:33.347913 8542 consensus_queue.cc:260] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:33.349624 8542 ts_tablet_manager.cc:1398] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240502 14:17:33.350654 7199 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-2f9a5a87d95a4657a2e8756962939ba2 received from {username='slave'} at 127.21.58.68:49317
I20240502 14:17:33.350807 7199 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-2f9a5a87d95a4657a2e8756962939ba2 on tablet 2f9a5a87d95a4657a2e8756962939ba2 with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:33.362730 8559 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:33.363220 8559 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: Bootstrap complete.
I20240502 14:17:33.363579 8559 ts_tablet_manager.cc:1367] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: Time spent bootstrapping tablet: real 0.055s user 0.044s sys 0.012s
I20240502 14:17:33.367818 8559 raft_consensus.cc:348] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:33.368185 8559 raft_consensus.cc:729] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Initialized, Role: LEARNER
I20240502 14:17:33.368700 8559 consensus_queue.cc:260] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:33.370007 8559 ts_tablet_manager.cc:1398] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:17:33.371017 7199 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 952df5ac1283460489e81cdb345007b7-67c65d90099b47f0859ce598e7e727c6 received from {username='slave'} at 127.21.58.69:35189
I20240502 14:17:33.371198 7199 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 952df5ac1283460489e81cdb345007b7-67c65d90099b47f0859ce598e7e727c6 on tablet 67c65d90099b47f0859ce598e7e727c6 with peer 952df5ac1283460489e81cdb345007b7
I20240502 14:17:33.404146 8559 ts_tablet_manager.cc:902] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: Initiating tablet copy from peer 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:33.405244 8559 tablet_copy_client.cc:308] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40049
I20240502 14:17:33.410727 7465 tablet_copy_service.cc:140] P 699831d1e03d4a109eb19f69687deac2: Received BeginTabletCopySession request for tablet 41104faa3b9a403e8e2316f8c40d4036 from peer 952df5ac1283460489e81cdb345007b7 ({username='slave'} at 127.21.58.69:36331)
I20240502 14:17:33.410931 7465 tablet_copy_service.cc:161] P 699831d1e03d4a109eb19f69687deac2: Beginning new tablet copy session on tablet 41104faa3b9a403e8e2316f8c40d4036 from peer 952df5ac1283460489e81cdb345007b7 at {username='slave'} at 127.21.58.69:36331: session id = 952df5ac1283460489e81cdb345007b7-41104faa3b9a403e8e2316f8c40d4036
I20240502 14:17:33.413628 7465 tablet_copy_source_session.cc:205] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:33.415210 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41104faa3b9a403e8e2316f8c40d4036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:33.419716 8559 tablet_copy_client.cc:783] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:33.420012 8559 tablet_copy_client.cc:647] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:33.421967 8559 tablet_copy_client.cc:523] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:33.422130 8542 ts_tablet_manager.cc:902] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:33.424623 8559 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: Bootstrap starting.
I20240502 14:17:33.427465 8542 tablet_copy_client.cc:308] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:33.428210 7199 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet c4830003d84f487d80b7744ca03c1fbe from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:49317)
I20240502 14:17:33.428386 7199 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet c4830003d84f487d80b7744ca03c1fbe from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:49317: session id = 1b8d0ccec3714c3694d8285fcceb4448-c4830003d84f487d80b7744ca03c1fbe
I20240502 14:17:33.431385 7199 tablet_copy_source_session.cc:205] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:33.432526 8542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4830003d84f487d80b7744ca03c1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:33.436579 8542 tablet_copy_client.cc:783] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:33.436792 8542 tablet_copy_client.cc:647] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:33.438273 8542 tablet_copy_client.cc:523] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:33.440482 8542 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:33.459980 8559 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:33.460491 8559 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: Bootstrap complete.
I20240502 14:17:33.460781 8559 ts_tablet_manager.cc:1367] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: Time spent bootstrapping tablet: real 0.036s user 0.032s sys 0.004s
I20240502 14:17:33.461539 8559 raft_consensus.cc:348] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:33.461848 8559 raft_consensus.cc:729] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Initialized, Role: LEARNER
I20240502 14:17:33.462045 8559 consensus_queue.cc:260] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:33.462541 8559 ts_tablet_manager.cc:1398] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:17:33.463354 7465 tablet_copy_service.cc:342] P 699831d1e03d4a109eb19f69687deac2: Request end of tablet copy session 952df5ac1283460489e81cdb345007b7-41104faa3b9a403e8e2316f8c40d4036 received from {username='slave'} at 127.21.58.69:36331
I20240502 14:17:33.463522 7465 tablet_copy_service.cc:433] P 699831d1e03d4a109eb19f69687deac2: ending tablet copy session 952df5ac1283460489e81cdb345007b7-41104faa3b9a403e8e2316f8c40d4036 on tablet 41104faa3b9a403e8e2316f8c40d4036 with peer 952df5ac1283460489e81cdb345007b7
I20240502 14:17:33.474828 8542 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:33.475245 8542 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:33.475488 8542 ts_tablet_manager.cc:1367] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.011s
I20240502 14:17:33.476219 8542 raft_consensus.cc:348] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:33.476433 8542 raft_consensus.cc:729] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:33.476622 8542 consensus_queue.cc:260] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:33.477241 8542 ts_tablet_manager.cc:1398] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:17:33.477895 7199 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-c4830003d84f487d80b7744ca03c1fbe received from {username='slave'} at 127.21.58.68:49317
I20240502 14:17:33.478041 7199 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-c4830003d84f487d80b7744ca03c1fbe on tablet c4830003d84f487d80b7744ca03c1fbe with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:33.482991 8542 ts_tablet_manager.cc:902] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565)
I20240502 14:17:33.484759 8542 tablet_copy_client.cc:308] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:36565
I20240502 14:17:33.491243 7332 tablet_copy_service.cc:140] P e230e4b943ef49059c6a55ad1e667a02: Received BeginTabletCopySession request for tablet d0cd8c0d16f844edb70cec8092dd7f05 from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:58981)
I20240502 14:17:33.491492 7332 tablet_copy_service.cc:161] P e230e4b943ef49059c6a55ad1e667a02: Beginning new tablet copy session on tablet d0cd8c0d16f844edb70cec8092dd7f05 from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:58981: session id = 1b8d0ccec3714c3694d8285fcceb4448-d0cd8c0d16f844edb70cec8092dd7f05
I20240502 14:17:33.494827 7332 tablet_copy_source_session.cc:205] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:33.496537 8542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cd8c0d16f844edb70cec8092dd7f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:33.501093 8542 tablet_copy_client.cc:783] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:33.501834 8542 tablet_copy_client.cc:647] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:33.506564 8542 tablet_copy_client.cc:523] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:33.511474 8542 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:33.518151 8559 ts_tablet_manager.cc:902] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: Initiating tablet copy from peer 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:33.523988 8559 tablet_copy_client.cc:308] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40049
I20240502 14:17:33.527222 7465 tablet_copy_service.cc:140] P 699831d1e03d4a109eb19f69687deac2: Received BeginTabletCopySession request for tablet 3293190336514711801388ea1e1a1583 from peer 952df5ac1283460489e81cdb345007b7 ({username='slave'} at 127.21.58.69:36331)
I20240502 14:17:33.527398 7465 tablet_copy_service.cc:161] P 699831d1e03d4a109eb19f69687deac2: Beginning new tablet copy session on tablet 3293190336514711801388ea1e1a1583 from peer 952df5ac1283460489e81cdb345007b7 at {username='slave'} at 127.21.58.69:36331: session id = 952df5ac1283460489e81cdb345007b7-3293190336514711801388ea1e1a1583
I20240502 14:17:33.530310 7465 tablet_copy_source_session.cc:205] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:33.531687 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3293190336514711801388ea1e1a1583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:33.537237 8559 tablet_copy_client.cc:783] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:33.537493 8559 tablet_copy_client.cc:647] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:33.539397 8559 tablet_copy_client.cc:523] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:33.541894 8559 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: Bootstrap starting.
I20240502 14:17:33.561110 8542 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:33.561610 8542 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:33.561849 8542 ts_tablet_manager.cc:1367] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.051s user 0.044s sys 0.004s
I20240502 14:17:33.562667 8542 raft_consensus.cc:348] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:33.562886 8542 raft_consensus.cc:729] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:33.563143 8542 consensus_queue.cc:260] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:33.563935 8542 ts_tablet_manager.cc:1398] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:33.564713 7332 tablet_copy_service.cc:342] P e230e4b943ef49059c6a55ad1e667a02: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-d0cd8c0d16f844edb70cec8092dd7f05 received from {username='slave'} at 127.21.58.68:58981
I20240502 14:17:33.564884 7332 tablet_copy_service.cc:433] P e230e4b943ef49059c6a55ad1e667a02: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-d0cd8c0d16f844edb70cec8092dd7f05 on tablet d0cd8c0d16f844edb70cec8092dd7f05 with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:33.576603 8559 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:33.576946 8559 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: Bootstrap complete.
I20240502 14:17:33.577133 8559 ts_tablet_manager.cc:1367] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.008s
I20240502 14:17:33.577687 8559 raft_consensus.cc:348] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:33.577862 8559 raft_consensus.cc:729] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Initialized, Role: LEARNER
I20240502 14:17:33.578009 8559 consensus_queue.cc:260] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:33.578403 8559 ts_tablet_manager.cc:1398] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:17:33.579128 7465 tablet_copy_service.cc:342] P 699831d1e03d4a109eb19f69687deac2: Request end of tablet copy session 952df5ac1283460489e81cdb345007b7-3293190336514711801388ea1e1a1583 received from {username='slave'} at 127.21.58.69:36331
I20240502 14:17:33.579250 7465 tablet_copy_service.cc:433] P 699831d1e03d4a109eb19f69687deac2: ending tablet copy session 952df5ac1283460489e81cdb345007b7-3293190336514711801388ea1e1a1583 on tablet 3293190336514711801388ea1e1a1583 with peer 952df5ac1283460489e81cdb345007b7
I20240502 14:17:33.645210 8242 raft_consensus.cc:1203] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:33.732977 8111 raft_consensus.cc:1203] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:33.875514 8111 raft_consensus.cc:1203] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:33.894593 8111 raft_consensus.cc:1203] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:33.945082 8242 raft_consensus.cc:1203] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:34.074313 8242 raft_consensus.cc:1203] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:17:34.090607 8620 raft_consensus.cc:1050] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:34.091436 8620 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:34.093607 7442 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.093609 8242 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.093716 7311 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.094172 8620 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.094535 8686 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.094805 7895 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.098634 7895 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.099373 7311 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.099330 8242 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.103224 7442 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.103669 7034 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.111034 7178 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:34.112610 8242 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.112701 7311 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.113094 7895 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.113389 8686 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.115269 8686 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.115842 7311 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.118754 7019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f9a5a87d95a4657a2e8756962939ba2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:34.119257 8242 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.119290 7026 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.123190 7422 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a5a87d95a4657a2e8756962939ba2 with delete_type TABLET_DATA_TOMBSTONED (TS 699831d1e03d4a109eb19f69687deac2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37414
I20240502 14:17:34.131362 8695 tablet_replica.cc:330] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: stopping tablet replica
I20240502 14:17:34.131799 8695 raft_consensus.cc:2229] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:34.132190 8695 raft_consensus.cc:2258] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:34.133459 8695 ts_tablet_manager.cc:1875] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:34.137614 8695 ts_tablet_manager.cc:1888] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:34.137746 8695 log.cc:1196] T 2f9a5a87d95a4657a2e8756962939ba2 P 699831d1e03d4a109eb19f69687deac2: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/2f9a5a87d95a4657a2e8756962939ba2
I20240502 14:17:34.138568 7021 catalog_manager.cc:4830] TS 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049): tablet 2f9a5a87d95a4657a2e8756962939ba2 (table rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]) successfully deleted
I20240502 14:17:34.219002 8686 raft_consensus.cc:1050] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 952df5ac1283460489e81cdb345007b7 to VOTER
I20240502 14:17:34.219673 8686 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.221791 7442 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.221834 7311 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.221901 8111 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.222368 7895 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.222690 8621 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.223023 8690 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.225960 8621 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.227397 8111 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.229034 7311 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.229653 7442 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.230719 7035 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.315732 7178 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6"
dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45488
I20240502 14:17:34.315986 7178 raft_consensus.cc:595] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:34.329134 8652 raft_consensus.cc:1050] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: attempting to promote NON_VOTER 952df5ac1283460489e81cdb345007b7 to VOTER
I20240502 14:17:34.329955 8652 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.331385 8471 raft_consensus.cc:1050] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: attempting to promote NON_VOTER 952df5ac1283460489e81cdb345007b7 to VOTER
I20240502 14:17:34.332221 8471 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.334777 8111 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.334795 7312 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.335947 8598 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.336328 7178 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.337651 8652 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.338186 8110 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.338897 8747 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.339126 7312 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.339676 8748 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.340029 8652 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.340409 7179 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.341301 8748 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.360723 8598 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.367895 8621 raft_consensus.cc:1050] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:34.369074 8621 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:34.369796 7026 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.369956 8110 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.371448 7177 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.380952 8242 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.381371 7442 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.382119 8620 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.382876 8620 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.386586 7312 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.387297 8621 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.387329 8748 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.388092 7310 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.392700 7178 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.393081 8110 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.394804 7311 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.396036 8621 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.396389 7035 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 4 to 5, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.401386 8242 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.402174 7443 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.403755 7441 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.404227 8110 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.405339 7442 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.405668 8110 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.405709 8598 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.405891 7309 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.406519 8598 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.407711 8652 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.408704 8652 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.409965 8111 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.410362 7026 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.411728 7311 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.412876 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41104faa3b9a403e8e2316f8c40d4036 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:34.413950 8747 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.414448 8747 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.416810 8110 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.417454 7309 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.417829 7026 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.418383 7310 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.422925 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3293190336514711801388ea1e1a1583 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:34.425786 7311 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.426577 7178 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:34.427549 7030 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 reported cstate change: config changed from index 5 to 6, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:34.429003 7309 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.430269 7158 tablet_service.cc:1514] Processing DeleteTablet for tablet 41104faa3b9a403e8e2316f8c40d4036 with delete_type TABLET_DATA_TOMBSTONED (TS f7ad4b3a72f04f259caff4473296ae72 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45220
I20240502 14:17:34.431337 8771 tablet_replica.cc:330] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: stopping tablet replica
I20240502 14:17:34.431912 8771 raft_consensus.cc:2229] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:34.432236 7157 tablet_service.cc:1514] Processing DeleteTablet for tablet 3293190336514711801388ea1e1a1583 with delete_type TABLET_DATA_TOMBSTONED (TS f7ad4b3a72f04f259caff4473296ae72 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45220
I20240502 14:17:34.432337 8771 raft_consensus.cc:2258] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:34.432957 8242 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.433283 8621 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.433533 8690 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.434319 8771 ts_tablet_manager.cc:1875] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:34.435851 8690 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.440310 7019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4830003d84f487d80b7744ca03c1fbe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:34.441468 7309 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.442087 8771 ts_tablet_manager.cc:1888] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:34.441486 8242 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.442306 8771 log.cc:1196] T 41104faa3b9a403e8e2316f8c40d4036 P f7ad4b3a72f04f259caff4473296ae72: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/41104faa3b9a403e8e2316f8c40d4036
I20240502 14:17:34.443552 8771 tablet_replica.cc:330] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: stopping tablet replica
I20240502 14:17:34.443557 7019 catalog_manager.cc:4830] TS f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021): tablet 41104faa3b9a403e8e2316f8c40d4036 (table rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]) successfully deleted
I20240502 14:17:34.443853 8771 raft_consensus.cc:2229] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:34.444234 8771 raft_consensus.cc:2258] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:34.445235 8771 ts_tablet_manager.cc:1875] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:34.445894 7035 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.451606 7422 tablet_service.cc:1514] Processing DeleteTablet for tablet c4830003d84f487d80b7744ca03c1fbe with delete_type TABLET_DATA_TOMBSTONED (TS 699831d1e03d4a109eb19f69687deac2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37414
I20240502 14:17:34.451898 8695 tablet_replica.cc:330] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: stopping tablet replica
I20240502 14:17:34.452070 8695 raft_consensus.cc:2229] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:34.452332 8695 raft_consensus.cc:2258] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:34.452576 8771 ts_tablet_manager.cc:1888] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:34.452694 8771 log.cc:1196] T 3293190336514711801388ea1e1a1583 P f7ad4b3a72f04f259caff4473296ae72: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/3293190336514711801388ea1e1a1583
I20240502 14:17:34.453112 8695 ts_tablet_manager.cc:1875] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:34.457036 7019 catalog_manager.cc:4830] TS f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021): tablet 3293190336514711801388ea1e1a1583 (table rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]) successfully deleted
I20240502 14:17:34.457852 8695 ts_tablet_manager.cc:1888] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:34.457974 8695 log.cc:1196] T c4830003d84f487d80b7744ca03c1fbe P 699831d1e03d4a109eb19f69687deac2: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/c4830003d84f487d80b7744ca03c1fbe
I20240502 14:17:34.458707 7021 catalog_manager.cc:4830] TS 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049): tablet c4830003d84f487d80b7744ca03c1fbe (table rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]) successfully deleted
I20240502 14:17:34.465031 7178 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6"
dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45520
I20240502 14:17:34.465191 7178 raft_consensus.cc:595] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:34.503854 7178 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6"
dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45540
I20240502 14:17:34.504027 7178 raft_consensus.cc:595] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:34.531478 7178 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6"
dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45568
I20240502 14:17:34.531706 7178 raft_consensus.cc:595] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:34.610442 8821 raft_consensus.cc:1050] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:34.611240 8821 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:34.613406 8242 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.613544 7442 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.613777 7178 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:34.614004 8823 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.614353 8821 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.614686 8684 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:34.617810 8823 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.618657 7442 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.621482 8242 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.622550 7035 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e230e4b943ef49059c6a55ad1e667a02" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.635344 7178 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.638075 7309 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:34.639902 8242 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.640324 7177 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:34.640399 8823 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.640797 8684 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:34.642637 8684 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.646256 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0cd8c0d16f844edb70cec8092dd7f05 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:34.647168 8242 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.648120 7178 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:34.649303 7026 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 5 to 6, VOTER 699831d1e03d4a109eb19f69687deac2 (127.21.58.67) evicted. New cstate: current_term: 1 leader_uuid: "e230e4b943ef49059c6a55ad1e667a02" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:34.654386 7422 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cd8c0d16f844edb70cec8092dd7f05 with delete_type TABLET_DATA_TOMBSTONED (TS 699831d1e03d4a109eb19f69687deac2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37414
I20240502 14:17:34.654738 8695 tablet_replica.cc:330] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: stopping tablet replica
I20240502 14:17:34.654937 8695 raft_consensus.cc:2229] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:34.655299 8695 raft_consensus.cc:2258] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:34.656136 8695 ts_tablet_manager.cc:1875] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:34.660357 8695 ts_tablet_manager.cc:1888] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:17:34.660452 8695 log.cc:1196] T d0cd8c0d16f844edb70cec8092dd7f05 P 699831d1e03d4a109eb19f69687deac2: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-2/wal/wals/d0cd8c0d16f844edb70cec8092dd7f05
I20240502 14:17:34.660974 7021 catalog_manager.cc:4830] TS 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049): tablet d0cd8c0d16f844edb70cec8092dd7f05 (table rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]) successfully deleted
I20240502 14:17:34.757594 7178 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6"
dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39230
I20240502 14:17:34.757781 7178 raft_consensus.cc:595] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:34.762142 8621 raft_consensus.cc:979] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: : Instructing follower 699831d1e03d4a109eb19f69687deac2 to start an election
I20240502 14:17:34.762339 8621 raft_consensus.cc:1067] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Signalling peer 699831d1e03d4a109eb19f69687deac2 to start an election
I20240502 14:17:34.763348 7442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6"
dest_uuid: "699831d1e03d4a109eb19f69687deac2"
from {username='slave'} at 127.21.58.65:36815
I20240502 14:17:34.763631 7442 raft_consensus.cc:482] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:34.763742 7442 raft_consensus.cc:3037] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:34.765645 7442 raft_consensus.cc:504] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.766335 7442 leader_election.cc:288] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 2 election: Requested vote from peers f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021), e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127)
I20240502 14:17:34.766623 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
I20240502 14:17:34.766834 7178 raft_consensus.cc:3032] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:34.766939 7178 raft_consensus.cc:729] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7ad4b3a72f04f259caff4473296ae72, State: Running, Role: LEADER
I20240502 14:17:34.766888 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:34.767064 8111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c65d90099b47f0859ce598e7e727c6" candidate_uuid: "699831d1e03d4a109eb19f69687deac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "952df5ac1283460489e81cdb345007b7"
I20240502 14:17:34.767413 7309 raft_consensus.cc:3037] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:34.767259 7178 consensus_queue.cc:260] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.767433 8111 raft_consensus.cc:3037] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:34.767791 7178 raft_consensus.cc:3037] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:34.769273 7309 raft_consensus.cc:2454] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 2.
I20240502 14:17:34.770406 8111 raft_consensus.cc:2454] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 2.
I20240502 14:17:34.770464 7178 raft_consensus.cc:2454] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699831d1e03d4a109eb19f69687deac2 in term 2.
I20240502 14:17:34.770860 7376 leader_election.cc:302] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 699831d1e03d4a109eb19f69687deac2, e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:34.771190 8471 raft_consensus.cc:2781] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:34.771401 8471 raft_consensus.cc:686] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 LEADER]: Becoming Leader. State: Replica: 699831d1e03d4a109eb19f69687deac2, State: Running, Role: LEADER
I20240502 14:17:34.771713 8471 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:34.774693 7034 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: term changed from 1 to 2, leader changed from f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) to 699831d1e03d4a109eb19f69687deac2 (127.21.58.67). New cstate: current_term: 2 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:35.184427 8111 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:35.185014 8853 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:35.255687 7309 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:35.256237 8853 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:35.260605 7178 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:35.261286 8652 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:35.263690 7442 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:35.265659 7309 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:35.266253 8652 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:35.268316 8110 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:35.269218 8652 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:35.270637 8853 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:35.271405 7309 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:35.271945 8110 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:35.272460 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67c65d90099b47f0859ce598e7e727c6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:35.273862 7035 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 5 to 7, VOTER f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65) evicted. New cstate: current_term: 2 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:35.277727 7157 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c65d90099b47f0859ce598e7e727c6 with delete_type TABLET_DATA_TOMBSTONED (TS f7ad4b3a72f04f259caff4473296ae72 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45220
I20240502 14:17:35.278404 8866 tablet_replica.cc:330] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: stopping tablet replica
I20240502 14:17:35.278607 8866 raft_consensus.cc:2229] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:35.278823 8866 raft_consensus.cc:2258] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:35.279631 8866 ts_tablet_manager.cc:1875] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:35.283787 8866 ts_tablet_manager.cc:1888] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:17:35.283906 8866 log.cc:1196] T 67c65d90099b47f0859ce598e7e727c6 P f7ad4b3a72f04f259caff4473296ae72: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-0/wal/wals/67c65d90099b47f0859ce598e7e727c6
I20240502 14:17:35.284518 7019 catalog_manager.cc:4830] TS f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021): tablet 67c65d90099b47f0859ce598e7e727c6 (table rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]) successfully deleted
I20240502 14:17:35.490098 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:35.510892 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:35.518910 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:35.521247 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:35.540879 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:35.693789 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:35.694926 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:35.696411 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:35.698786 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:35.719774 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:35.867264 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:35.867609 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:35.868229 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:35.881181 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:35.882305 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:36.443910 7035 ts_manager.cc:292] Set tserver state for e230e4b943ef49059c6a55ad1e667a02 to MAINTENANCE_MODE
I20240502 14:17:36.974119 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:36.991292 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:36.991825 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:36.993341 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:37.006084 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:37.165879 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:37.171679 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:37.175936 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:37.178498 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:37.208609 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:37.366443 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:37.367385 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:37.383680 8091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:37.389868 7292 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:37.417024 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:37.471884 7178 consensus_queue.cc:237] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:37.473858 7442 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:37.473904 7309 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:37.474349 8686 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:37.474640 9098 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:37.477737 8686 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:37.478349 7309 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:37.478405 7442 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:37.482681 7034 catalog_manager.cc:5484] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:37.490607 7178 consensus_queue.cc:237] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
W20240502 14:17:37.493353 7113 consensus_peers.cc:485] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ccb2c50626e4d3790f1cc006a4a7446. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:37.493492 7442 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:37.494063 9098 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:37.495589 7309 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:37.496388 9101 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:37.496996 8686 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:37.497819 7442 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:37.498239 7309 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:37.501173 7019 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1ccb2c50626e4d3790f1cc006a4a7446 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:37.502447 7035 catalog_manager.cc:5484] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:37.503783 7178 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:37.505878 8242 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:37.506412 9098 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:37.506599 7309 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:37.507189 9098 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:37.509033 9101 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:37.510274 7309 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:37.512212 8242 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:37.513307 7026 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:37.523691 7178 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
W20240502 14:17:37.526676 7113 consensus_peers.cc:485] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 -> Peer 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127): Couldn't send request to peer 952df5ac1283460489e81cdb345007b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4830003d84f487d80b7744ca03c1fbe. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:37.526978 7309 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:37.527120 8242 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:37.527683 9101 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:37.528132 9101 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:37.529817 9101 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:37.530726 8242 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:37.531495 7309 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:37.533900 7019 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c4830003d84f487d80b7744ca03c1fbe with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:37.536634 7026 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:37.545666 7178 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:37.547317 7309 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:37.547449 8242 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:37.547891 9101 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:37.548149 9098 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:37.550438 8686 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:37.551112 7309 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:37.551121 8242 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:37.555161 7026 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:37.561256 7178 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:37.563144 8242 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:37.563787 9098 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:37.564049 7309 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:37.564597 9098 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:17:37.564888 7113 consensus_peers.cc:485] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 -> Peer 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127): Couldn't send request to peer 952df5ac1283460489e81cdb345007b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f9a5a87d95a4657a2e8756962939ba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:37.566349 8686 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:37.566978 8242 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:37.568034 7309 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:37.571381 7034 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:37.571897 7019 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2f9a5a87d95a4657a2e8756962939ba2 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:37.980741 9153 ts_tablet_manager.cc:902] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:37.981860 9156 ts_tablet_manager.cc:902] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:37.982511 9153 tablet_copy_client.cc:308] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:37.983211 7199 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet 1ccb2c50626e4d3790f1cc006a4a7446 from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:49317)
I20240502 14:17:37.983422 7199 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet 1ccb2c50626e4d3790f1cc006a4a7446 from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:49317: session id = 1b8d0ccec3714c3694d8285fcceb4448-1ccb2c50626e4d3790f1cc006a4a7446
I20240502 14:17:37.983652 9156 tablet_copy_client.cc:308] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:37.984359 7198 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet 2f9a5a87d95a4657a2e8756962939ba2 from peer 952df5ac1283460489e81cdb345007b7 ({username='slave'} at 127.21.58.69:35189)
I20240502 14:17:37.984582 7198 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet 2f9a5a87d95a4657a2e8756962939ba2 from peer 952df5ac1283460489e81cdb345007b7 at {username='slave'} at 127.21.58.69:35189: session id = 952df5ac1283460489e81cdb345007b7-2f9a5a87d95a4657a2e8756962939ba2
I20240502 14:17:37.985229 9163 ts_tablet_manager.cc:902] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:37.986302 7199 tablet_copy_source_session.cc:205] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:37.987579 9153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccb2c50626e4d3790f1cc006a4a7446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:37.988441 7198 tablet_copy_source_session.cc:205] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:37.989543 9163 tablet_copy_client.cc:308] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:37.989789 9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5a87d95a4657a2e8756962939ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:37.990144 7198 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet c4830003d84f487d80b7744ca03c1fbe from peer 952df5ac1283460489e81cdb345007b7 ({username='slave'} at 127.21.58.69:35189)
I20240502 14:17:37.990278 7198 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet c4830003d84f487d80b7744ca03c1fbe from peer 952df5ac1283460489e81cdb345007b7 at {username='slave'} at 127.21.58.69:35189: session id = 952df5ac1283460489e81cdb345007b7-c4830003d84f487d80b7744ca03c1fbe
I20240502 14:17:37.992182 7198 tablet_copy_source_session.cc:205] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:37.993156 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4830003d84f487d80b7744ca03c1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:37.993402 9153 tablet_copy_client.cc:783] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:37.993636 9153 tablet_copy_client.cc:647] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:37.995849 9153 tablet_copy_client.cc:523] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:37.996440 9156 tablet_copy_client.cc:783] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:37.996680 9156 tablet_copy_client.cc:647] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:37.998181 9156 tablet_copy_client.cc:523] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:37.998457 9153 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:37.999161 9163 tablet_copy_client.cc:783] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:37.999421 9163 tablet_copy_client.cc:647] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:38.001150 9156 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: Bootstrap starting.
I20240502 14:17:38.002283 9163 tablet_copy_client.cc:523] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:38.005599 9163 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: Bootstrap starting.
I20240502 14:17:38.048946 9153 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:38.049399 9153 tablet_bootstrap.cc:492] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:38.049685 9153 ts_tablet_manager.cc:1367] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.051s user 0.044s sys 0.005s
I20240502 14:17:38.050467 9153 raft_consensus.cc:348] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:38.050738 9153 raft_consensus.cc:729] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:38.050956 9153 consensus_queue.cc:260] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:38.051522 9153 ts_tablet_manager.cc:1398] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:17:38.052285 7198 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-1ccb2c50626e4d3790f1cc006a4a7446 received from {username='slave'} at 127.21.58.68:49317
I20240502 14:17:38.052417 7198 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-1ccb2c50626e4d3790f1cc006a4a7446 on tablet 1ccb2c50626e4d3790f1cc006a4a7446 with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:38.060748 9156 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:38.061213 9156 tablet_bootstrap.cc:492] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: Bootstrap complete.
I20240502 14:17:38.061501 9156 ts_tablet_manager.cc:1367] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: Time spent bootstrapping tablet: real 0.060s user 0.044s sys 0.009s
I20240502 14:17:38.062268 9156 raft_consensus.cc:348] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:38.062579 9156 raft_consensus.cc:729] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Initialized, Role: LEARNER
I20240502 14:17:38.062852 9163 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:38.062800 9156 consensus_queue.cc:260] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:38.063305 9163 tablet_bootstrap.cc:492] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: Bootstrap complete.
I20240502 14:17:38.063572 9163 ts_tablet_manager.cc:1367] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: Time spent bootstrapping tablet: real 0.058s user 0.049s sys 0.005s
I20240502 14:17:38.064380 9156 ts_tablet_manager.cc:1398] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240502 14:17:38.064929 9163 raft_consensus.cc:348] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:38.065054 7198 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 952df5ac1283460489e81cdb345007b7-2f9a5a87d95a4657a2e8756962939ba2 received from {username='slave'} at 127.21.58.69:35189
I20240502 14:17:38.065171 7198 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 952df5ac1283460489e81cdb345007b7-2f9a5a87d95a4657a2e8756962939ba2 on tablet 2f9a5a87d95a4657a2e8756962939ba2 with peer 952df5ac1283460489e81cdb345007b7
I20240502 14:17:38.065188 9163 raft_consensus.cc:729] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Initialized, Role: LEARNER
I20240502 14:17:38.065387 9163 consensus_queue.cc:260] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:38.065901 9163 ts_tablet_manager.cc:1398] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:17:38.069422 7198 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 952df5ac1283460489e81cdb345007b7-c4830003d84f487d80b7744ca03c1fbe received from {username='slave'} at 127.21.58.69:35189
I20240502 14:17:38.069535 7198 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 952df5ac1283460489e81cdb345007b7-c4830003d84f487d80b7744ca03c1fbe on tablet c4830003d84f487d80b7744ca03c1fbe with peer 952df5ac1283460489e81cdb345007b7
I20240502 14:17:38.415306 8242 raft_consensus.cc:1203] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:17:38.457607 8110 raft_consensus.cc:1203] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:17:38.500638 8110 raft_consensus.cc:1203] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:17:38.888479 9101 raft_consensus.cc:1050] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 952df5ac1283460489e81cdb345007b7 to VOTER
I20240502 14:17:38.889134 9101 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:38.891346 8242 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:38.891436 8110 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:38.891477 7311 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:38.891934 9233 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:38.892273 9101 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:38.892566 8686 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:38.895480 9101 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.896410 8110 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.898260 8242 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.900450 7035 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 reported cstate change: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.902299 7311 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.919689 7178 consensus_queue.cc:237] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:38.921418 8242 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:38.921577 8110 raft_consensus.cc:1261] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:38.922073 9098 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:38.922360 8686 consensus_queue.cc:1035] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:38.922541 9101 raft_consensus.cc:1050] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 952df5ac1283460489e81cdb345007b7 to VOTER
I20240502 14:17:38.923619 9101 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:38.924327 8686 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.925613 8242 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.927366 8111 raft_consensus.cc:2932] T c4830003d84f487d80b7744ca03c1fbe P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.928422 7026 catalog_manager.cc:5484] T c4830003d84f487d80b7744ca03c1fbe P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:38.929016 8110 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:38.929384 7019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4830003d84f487d80b7744ca03c1fbe with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:38.930332 8242 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:38.930706 9098 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:38.931025 9101 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:38.933161 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet c4830003d84f487d80b7744ca03c1fbe with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:38.934262 7311 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:38.934736 9101 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:38.937655 8686 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.938900 7311 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.938879 8242 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.941623 8110 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.942669 7026 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:38.948513 7178 consensus_queue.cc:237] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:38.949365 9244 tablet_replica.cc:330] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:38.950161 8110 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:38.950167 8242 raft_consensus.cc:1261] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:38.950203 9244 raft_consensus.cc:2229] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:38.950634 9101 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:38.950982 9244 raft_consensus.cc:2258] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:38.950987 9098 consensus_queue.cc:1035] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:38.952780 9244 ts_tablet_manager.cc:1875] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:38.953043 9101 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.954269 8241 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.955618 7019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f9a5a87d95a4657a2e8756962939ba2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:38.956068 8110 raft_consensus.cc:2932] T 2f9a5a87d95a4657a2e8756962939ba2 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.959740 7030 catalog_manager.cc:5484] T 2f9a5a87d95a4657a2e8756962939ba2 P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:38.963311 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a5a87d95a4657a2e8756962939ba2 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:38.967305 9244 ts_tablet_manager.cc:1888] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:17:38.967470 9244 log.cc:1196] T c4830003d84f487d80b7744ca03c1fbe P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/c4830003d84f487d80b7744ca03c1fbe
I20240502 14:17:38.968230 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet c4830003d84f487d80b7744ca03c1fbe (table rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]) successfully deleted
I20240502 14:17:38.968298 9244 tablet_replica.cc:330] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:38.968456 9244 raft_consensus.cc:2229] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:38.968775 9244 raft_consensus.cc:2258] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:38.969779 9244 ts_tablet_manager.cc:1875] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:38.975359 9244 ts_tablet_manager.cc:1888] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:17:38.975494 9244 log.cc:1196] T 2f9a5a87d95a4657a2e8756962939ba2 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f9a5a87d95a4657a2e8756962939ba2
I20240502 14:17:38.976131 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet 2f9a5a87d95a4657a2e8756962939ba2 (table rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]) successfully deleted
I20240502 14:17:38.990113 8686 raft_consensus.cc:1050] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:38.990945 8686 consensus_queue.cc:237] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:38.993172 7311 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:38.993248 8241 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:38.993315 7442 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:38.993808 8686 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:38.994089 9101 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:38.994316 9233 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:38.998919 9101 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:38.999502 8241 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:39.003584 7026 catalog_manager.cc:5484] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:39.004645 7442 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:39.008543 7178 consensus_queue.cc:237] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:39.011057 7442 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:39.011193 8241 raft_consensus.cc:1261] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:39.011696 8686 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:39.011965 9101 consensus_queue.cc:1035] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:39.013948 9233 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:39.014592 8241 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:39.014696 7442 raft_consensus.cc:2932] T 1ccb2c50626e4d3790f1cc006a4a7446 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:39.016451 7019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ccb2c50626e4d3790f1cc006a4a7446 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:39.017911 7030 catalog_manager.cc:5484] T 1ccb2c50626e4d3790f1cc006a4a7446 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:39.021179 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ccb2c50626e4d3790f1cc006a4a7446 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:39.021616 9244 tablet_replica.cc:330] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:39.021821 9244 raft_consensus.cc:2229] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:39.021991 9244 pending_rounds.cc:62] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Trying to abort 1 pending ops.
I20240502 14:17:39.022082 9244 pending_rounds.cc:68] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023245144024715264 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "1ccb2c50626e4d3790f1cc006a4a7446" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } } }
I20240502 14:17:39.022399 9244 raft_consensus.cc:2894] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 14:17:39.022531 9244 raft_consensus.cc:2866] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:17:39.022635 9244 raft_consensus.cc:2258] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:39.023303 9244 ts_tablet_manager.cc:1875] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:39.026813 9244 ts_tablet_manager.cc:1888] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:17:39.026906 9244 log.cc:1196] T 1ccb2c50626e4d3790f1cc006a4a7446 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/1ccb2c50626e4d3790f1cc006a4a7446
I20240502 14:17:39.027523 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet 1ccb2c50626e4d3790f1cc006a4a7446 (table rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]) successfully deleted
I20240502 14:17:39.208542 7442 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:39.210686 8110 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:39.210731 7311 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:39.211270 8971 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:39.211601 9272 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:39.214370 8971 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:39.215723 8110 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:39.217339 7311 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:39.218627 7034 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:39.223816 7442 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:39.226454 8110 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:39.226914 8971 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:39.227631 7311 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:39.229518 9272 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:39.230330 9272 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.231953 7178 consensus_queue.cc:237] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:39.233160 8111 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.234133 7021 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3293190336514711801388ea1e1a1583 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:39.236408 7309 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:39.236932 7442 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:39.237033 9101 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:39.237246 7030 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 reported cstate change: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.237850 9101 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:17:39.238090 7378 consensus_peers.cc:485] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3293190336514711801388ea1e1a1583. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:39.238597 7311 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.241269 9101 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:39.245357 7442 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:39.246136 7030 catalog_manager.cc:5484] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } }
I20240502 14:17:39.251372 7178 consensus_queue.cc:237] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
W20240502 14:17:39.255026 7113 consensus_peers.cc:485] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6af3a6191cd34b3dac8900eb0ef265e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:39.256582 7311 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:39.257318 7442 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:39.257596 9101 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:39.257823 9233 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:39.259442 7309 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:39.259756 9233 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.263823 7442 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.267145 7019 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6af3a6191cd34b3dac8900eb0ef265e2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:39.268134 7309 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.268327 7030 catalog_manager.cc:5484] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 4 to 5, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.496649 7442 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:39.498680 8110 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:39.498741 7309 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:39.499284 9275 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:39.499573 9272 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:39.502051 8971 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:39.502640 7309 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:39.502705 8110 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:39.506772 7030 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 7 to 8. New cstate: current_term: 2 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:39.511729 7442 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:39.513458 8110 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:17:39.513579 7309 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 14:17:39.513733 7378 consensus_peers.cc:485] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67c65d90099b47f0859ce598e7e727c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:39.514153 9275 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:39.514580 9275 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:39.516827 9275 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.517483 7309 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.517488 8110 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:39.520071 7021 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 67c65d90099b47f0859ce598e7e727c6 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:39.521396 7030 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 8 to 9, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 2 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:39.671437 9325 ts_tablet_manager.cc:902] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:39.672729 9325 tablet_copy_client.cc:308] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40049
I20240502 14:17:39.678082 7465 tablet_copy_service.cc:140] P 699831d1e03d4a109eb19f69687deac2: Received BeginTabletCopySession request for tablet 3293190336514711801388ea1e1a1583 from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:33195)
I20240502 14:17:39.678220 7465 tablet_copy_service.cc:161] P 699831d1e03d4a109eb19f69687deac2: Beginning new tablet copy session on tablet 3293190336514711801388ea1e1a1583 from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:33195: session id = 1b8d0ccec3714c3694d8285fcceb4448-3293190336514711801388ea1e1a1583
I20240502 14:17:39.680269 7465 tablet_copy_source_session.cc:205] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:39.681388 9325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3293190336514711801388ea1e1a1583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:39.685562 9325 tablet_copy_client.cc:783] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:39.685813 9325 tablet_copy_client.cc:647] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:39.687270 9325 tablet_copy_client.cc:523] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:39.689761 9325 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:39.719835 9325 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:39.720187 9325 tablet_bootstrap.cc:492] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:39.720412 9325 ts_tablet_manager.cc:1367] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.013s
I20240502 14:17:39.721024 9325 raft_consensus.cc:348] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:39.721218 9325 raft_consensus.cc:729] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:39.721402 9325 consensus_queue.cc:260] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:39.722175 9325 ts_tablet_manager.cc:1398] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:17:39.722864 7465 tablet_copy_service.cc:342] P 699831d1e03d4a109eb19f69687deac2: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-3293190336514711801388ea1e1a1583 received from {username='slave'} at 127.21.58.68:33195
I20240502 14:17:39.723021 7465 tablet_copy_service.cc:433] P 699831d1e03d4a109eb19f69687deac2: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-3293190336514711801388ea1e1a1583 on tablet 3293190336514711801388ea1e1a1583 with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:39.781976 9325 ts_tablet_manager.cc:902] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:39.783113 9325 tablet_copy_client.cc:308] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:39.783653 7198 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet 6af3a6191cd34b3dac8900eb0ef265e2 from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:49317)
I20240502 14:17:39.783779 7198 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet 6af3a6191cd34b3dac8900eb0ef265e2 from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:49317: session id = 1b8d0ccec3714c3694d8285fcceb4448-6af3a6191cd34b3dac8900eb0ef265e2
I20240502 14:17:39.785537 7198 tablet_copy_source_session.cc:205] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:39.786454 9325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af3a6191cd34b3dac8900eb0ef265e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:39.790289 9325 tablet_copy_client.cc:783] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:39.790478 9325 tablet_copy_client.cc:647] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:39.791946 9325 tablet_copy_client.cc:523] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:39.793957 9325 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:39.820820 9325 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:39.821190 9325 tablet_bootstrap.cc:492] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:39.821403 9325 ts_tablet_manager.cc:1367] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.004s
I20240502 14:17:39.822036 9325 raft_consensus.cc:348] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:39.822218 9325 raft_consensus.cc:729] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:39.822425 9325 consensus_queue.cc:260] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:39.822872 9325 ts_tablet_manager.cc:1398] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:17:39.823637 7198 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-6af3a6191cd34b3dac8900eb0ef265e2 received from {username='slave'} at 127.21.58.68:49317
I20240502 14:17:39.823768 7198 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-6af3a6191cd34b3dac8900eb0ef265e2 on tablet 6af3a6191cd34b3dac8900eb0ef265e2 with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:40.050436 9325 ts_tablet_manager.cc:902] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:40.051591 9325 tablet_copy_client.cc:308] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40049
I20240502 14:17:40.052204 7465 tablet_copy_service.cc:140] P 699831d1e03d4a109eb19f69687deac2: Received BeginTabletCopySession request for tablet 67c65d90099b47f0859ce598e7e727c6 from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:33195)
I20240502 14:17:40.052347 7465 tablet_copy_service.cc:161] P 699831d1e03d4a109eb19f69687deac2: Beginning new tablet copy session on tablet 67c65d90099b47f0859ce598e7e727c6 from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:33195: session id = 1b8d0ccec3714c3694d8285fcceb4448-67c65d90099b47f0859ce598e7e727c6
I20240502 14:17:40.054167 7465 tablet_copy_source_session.cc:205] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:40.055223 9325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c65d90099b47f0859ce598e7e727c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:40.059429 9325 tablet_copy_client.cc:783] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:40.059659 9325 tablet_copy_client.cc:647] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:40.061043 9325 tablet_copy_client.cc:523] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:40.063241 9325 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:40.092540 9325 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:40.092965 9325 tablet_bootstrap.cc:492] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:40.093218 9325 ts_tablet_manager.cc:1367] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.002s
I20240502 14:17:40.093856 9325 raft_consensus.cc:348] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:40.094053 9325 raft_consensus.cc:729] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:40.094254 9325 consensus_queue.cc:260] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:40.094729 9325 ts_tablet_manager.cc:1398] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:17:40.095446 7465 tablet_copy_service.cc:342] P 699831d1e03d4a109eb19f69687deac2: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-67c65d90099b47f0859ce598e7e727c6 received from {username='slave'} at 127.21.58.68:33195
I20240502 14:17:40.095582 7465 tablet_copy_service.cc:433] P 699831d1e03d4a109eb19f69687deac2: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-67c65d90099b47f0859ce598e7e727c6 on tablet 67c65d90099b47f0859ce598e7e727c6 with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:40.167935 8241 raft_consensus.cc:1203] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:17:40.274448 8241 raft_consensus.cc:1203] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:17:40.566455 8241 raft_consensus.cc:1203] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 14:17:40.772832 9275 raft_consensus.cc:1050] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:40.773550 9275 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:40.775683 8241 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:40.775734 8110 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:40.775789 7309 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:40.776291 9275 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:40.776575 9272 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:40.776886 9422 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:40.780194 9422 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.780699 8110 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.786244 8241 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.787416 7034 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:40.793148 7311 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.801431 7442 consensus_queue.cc:237] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:40.802878 8110 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:40.802883 8241 raft_consensus.cc:1261] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:40.803345 9272 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:40.803583 9275 consensus_queue.cc:1035] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:40.805389 9374 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.806017 8110 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.806025 8241 raft_consensus.cc:2932] T 3293190336514711801388ea1e1a1583 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.809491 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3293190336514711801388ea1e1a1583 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:40.810254 7035 catalog_manager.cc:5484] T 3293190336514711801388ea1e1a1583 P 952df5ac1283460489e81cdb345007b7 reported cstate change: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.814074 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet 3293190336514711801388ea1e1a1583 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:40.815318 9431 tablet_replica.cc:330] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:40.815493 9431 raft_consensus.cc:2229] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:40.815773 9431 raft_consensus.cc:2258] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:40.816468 9431 ts_tablet_manager.cc:1875] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:40.820963 9431 ts_tablet_manager.cc:1888] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:17:40.821105 9431 log.cc:1196] T 3293190336514711801388ea1e1a1583 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/3293190336514711801388ea1e1a1583
I20240502 14:17:40.821808 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet 3293190336514711801388ea1e1a1583 (table rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]) successfully deleted
I20240502 14:17:40.855988 9390 raft_consensus.cc:1050] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:40.856680 9390 consensus_queue.cc:237] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:40.858919 8241 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:40.859040 7442 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:40.859208 7311 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:40.859581 9390 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:40.859922 9433 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:40.860208 9368 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:40.864166 9433 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.864612 7311 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.868727 7026 catalog_manager.cc:5484] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.870234 8241 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.874464 7442 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.877821 7178 consensus_queue.cc:237] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:40.879618 8242 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:40.880138 9433 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:40.880456 7442 raft_consensus.cc:1261] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:40.882318 9390 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.882890 8242 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.883379 9368 consensus_queue.cc:1035] T 6af3a6191cd34b3dac8900eb0ef265e2 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:40.884877 7442 raft_consensus.cc:2932] T 6af3a6191cd34b3dac8900eb0ef265e2 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.886730 7019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6af3a6191cd34b3dac8900eb0ef265e2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:40.887795 7030 catalog_manager.cc:5484] T 6af3a6191cd34b3dac8900eb0ef265e2 P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:40.891472 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af3a6191cd34b3dac8900eb0ef265e2 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:40.891937 9431 tablet_replica.cc:330] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:40.892130 9431 raft_consensus.cc:2229] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:40.892341 9431 raft_consensus.cc:2258] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:40.893268 9431 ts_tablet_manager.cc:1875] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:40.898492 9431 ts_tablet_manager.cc:1888] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:17:40.898597 9431 log.cc:1196] T 6af3a6191cd34b3dac8900eb0ef265e2 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/6af3a6191cd34b3dac8900eb0ef265e2
I20240502 14:17:40.899299 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet 6af3a6191cd34b3dac8900eb0ef265e2 (table rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]) successfully deleted
I20240502 14:17:41.092702 9374 raft_consensus.cc:1050] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:41.093417 9374 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:41.095341 8242 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 LEARNER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:41.095592 7311 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:41.095655 8110 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:41.095984 9374 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:41.096272 9272 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:41.096554 9275 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:41.099970 9374 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.100576 8110 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.100826 7311 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.104779 7030 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 reported cstate change: config changed from index 9 to 10, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.104722 8242 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.111361 7442 consensus_queue.cc:237] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:41.112861 8242 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:17:41.112998 8110 raft_consensus.cc:1261] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:17:41.113381 9422 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:17:41.113615 9275 consensus_queue.cc:1035] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:17:41.115779 9374 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 699831d1e03d4a109eb19f69687deac2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.116466 8242 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.116475 8110 raft_consensus.cc:2932] T 67c65d90099b47f0859ce598e7e727c6 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.119918 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67c65d90099b47f0859ce598e7e727c6 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:41.122401 7035 catalog_manager.cc:5484] T 67c65d90099b47f0859ce598e7e727c6 P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 10 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.125952 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c65d90099b47f0859ce598e7e727c6 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:41.126294 9431 tablet_replica.cc:330] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:41.126492 9431 raft_consensus.cc:2229] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:41.127027 9431 raft_consensus.cc:2258] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:41.128356 9431 ts_tablet_manager.cc:1875] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:41.133265 9431 ts_tablet_manager.cc:1888] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 14:17:41.133375 9431 log.cc:1196] T 67c65d90099b47f0859ce598e7e727c6 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/67c65d90099b47f0859ce598e7e727c6
I20240502 14:17:41.133625 7178 consensus_queue.cc:237] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } }
I20240502 14:17:41.134006 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet 67c65d90099b47f0859ce598e7e727c6 (table rebalance_test_table_1 [id=9642c4050b7144da9073938ec4d0eea0]) successfully deleted
I20240502 14:17:41.135524 7311 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:41.135581 7442 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:41.136032 9433 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:41.136307 9368 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:41.138695 9433 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:41.139263 7311 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:41.141461 7442 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } }
I20240502 14:17:41.142560 7035 catalog_manager.cc:5484] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } }
I20240502 14:17:41.147346 7178 consensus_queue.cc:237] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:41.149266 7311 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:17:41.150141 7113 consensus_peers.cc:485] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 -> Peer 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127): Couldn't send request to peer 952df5ac1283460489e81cdb345007b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2c230f5dbc14ee886256e6710c817b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:41.150319 7442 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:41.150424 9390 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:41.151423 9390 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:41.153450 9368 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:41.154029 7311 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:41.156500 7442 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:41.158308 7019 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c2c230f5dbc14ee886256e6710c817b9 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:41.159986 7030 catalog_manager.cc:5484] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 4 to 5, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:41.160462 7441 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:41.163857 8110 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:41.164345 7311 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:41.165225 9422 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:41.165974 9374 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:41.168680 9422 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:41.169323 7311 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:41.176596 7035 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:41.182735 7441 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
W20240502 14:17:41.185031 7378 consensus_peers.cc:485] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 -> Peer 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889): Couldn't send request to peer 1b8d0ccec3714c3694d8285fcceb4448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41104faa3b9a403e8e2316f8c40d4036. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:41.185972 7311 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:41.187997 9374 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:41.191756 9374 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:41.193456 8111 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:41.194432 9275 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:41.195788 7021 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 41104faa3b9a403e8e2316f8c40d4036 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:41.196759 7309 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:41.202386 7035 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:17:41.203125 8110 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:41.211642 8110 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } } }
I20240502 14:17:41.451627 7309 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:41.453725 8242 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:41.453800 7178 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:41.454213 8821 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:41.454627 9495 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:41.457198 8821 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.457854 7178 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.458160 8242 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:41.461813 7035 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e230e4b943ef49059c6a55ad1e667a02" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:41.468586 7309 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:41.472041 8242 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:41.472200 7178 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:17:41.472723 9495 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:41.473032 8821 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:41.475607 9495 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:41.478931 7178 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
W20240502 14:17:41.479218 7246 consensus_peers.cc:485] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 -> Peer 952df5ac1283460489e81cdb345007b7 (127.21.58.69:34127): Couldn't send request to peer 952df5ac1283460489e81cdb345007b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0cd8c0d16f844edb70cec8092dd7f05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:41.479204 8242 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:41.483119 7030 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 7 to 8, NON_VOTER 952df5ac1283460489e81cdb345007b7 (127.21.58.69) added. New cstate: current_term: 1 leader_uuid: "e230e4b943ef49059c6a55ad1e667a02" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } } }
I20240502 14:17:41.483741 7021 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d0cd8c0d16f844edb70cec8092dd7f05 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:17:41.528820 9517 ts_tablet_manager.cc:902] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: Initiating tablet copy from peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021)
I20240502 14:17:41.530051 9517 tablet_copy_client.cc:308] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.65:37021
I20240502 14:17:41.530668 7198 tablet_copy_service.cc:140] P f7ad4b3a72f04f259caff4473296ae72: Received BeginTabletCopySession request for tablet c2c230f5dbc14ee886256e6710c817b9 from peer 952df5ac1283460489e81cdb345007b7 ({username='slave'} at 127.21.58.69:35189)
I20240502 14:17:41.530866 7198 tablet_copy_service.cc:161] P f7ad4b3a72f04f259caff4473296ae72: Beginning new tablet copy session on tablet c2c230f5dbc14ee886256e6710c817b9 from peer 952df5ac1283460489e81cdb345007b7 at {username='slave'} at 127.21.58.69:35189: session id = 952df5ac1283460489e81cdb345007b7-c2c230f5dbc14ee886256e6710c817b9
I20240502 14:17:41.533243 7198 tablet_copy_source_session.cc:205] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:41.534402 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c230f5dbc14ee886256e6710c817b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:41.538489 9517 tablet_copy_client.cc:783] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:41.538707 9517 tablet_copy_client.cc:647] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:41.540221 9517 tablet_copy_client.cc:523] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:41.542275 9517 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: Bootstrap starting.
I20240502 14:17:41.570576 9517 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:41.570966 9517 tablet_bootstrap.cc:492] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: Bootstrap complete.
I20240502 14:17:41.571234 9517 ts_tablet_manager.cc:1367] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.009s
I20240502 14:17:41.571830 9517 raft_consensus.cc:348] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:41.572047 9517 raft_consensus.cc:729] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Initialized, Role: LEARNER
I20240502 14:17:41.572221 9517 consensus_queue.cc:260] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:41.572701 9517 ts_tablet_manager.cc:1398] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:17:41.573529 7198 tablet_copy_service.cc:342] P f7ad4b3a72f04f259caff4473296ae72: Request end of tablet copy session 952df5ac1283460489e81cdb345007b7-c2c230f5dbc14ee886256e6710c817b9 received from {username='slave'} at 127.21.58.69:35189
I20240502 14:17:41.573688 7198 tablet_copy_service.cc:433] P f7ad4b3a72f04f259caff4473296ae72: ending tablet copy session 952df5ac1283460489e81cdb345007b7-c2c230f5dbc14ee886256e6710c817b9 on tablet c2c230f5dbc14ee886256e6710c817b9 with peer 952df5ac1283460489e81cdb345007b7
I20240502 14:17:41.801182 9538 ts_tablet_manager.cc:902] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: Initiating tablet copy from peer 699831d1e03d4a109eb19f69687deac2 (127.21.58.67:40049)
I20240502 14:17:41.802485 9538 tablet_copy_client.cc:308] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.67:40049
I20240502 14:17:41.803174 7465 tablet_copy_service.cc:140] P 699831d1e03d4a109eb19f69687deac2: Received BeginTabletCopySession request for tablet 41104faa3b9a403e8e2316f8c40d4036 from peer 1b8d0ccec3714c3694d8285fcceb4448 ({username='slave'} at 127.21.58.68:33195)
I20240502 14:17:41.803333 7465 tablet_copy_service.cc:161] P 699831d1e03d4a109eb19f69687deac2: Beginning new tablet copy session on tablet 41104faa3b9a403e8e2316f8c40d4036 from peer 1b8d0ccec3714c3694d8285fcceb4448 at {username='slave'} at 127.21.58.68:33195: session id = 1b8d0ccec3714c3694d8285fcceb4448-41104faa3b9a403e8e2316f8c40d4036
I20240502 14:17:41.805085 7465 tablet_copy_source_session.cc:205] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:41.806118 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41104faa3b9a403e8e2316f8c40d4036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:41.810452 9538 tablet_copy_client.cc:783] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:41.810667 9538 tablet_copy_client.cc:647] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:41.812161 9538 tablet_copy_client.cc:523] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:41.814298 9538 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap starting.
I20240502 14:17:41.844775 9538 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:41.845156 9538 tablet_bootstrap.cc:492] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: Bootstrap complete.
I20240502 14:17:41.845402 9538 ts_tablet_manager.cc:1367] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.004s
I20240502 14:17:41.846041 9538 raft_consensus.cc:348] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:41.846226 9538 raft_consensus.cc:729] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b8d0ccec3714c3694d8285fcceb4448, State: Initialized, Role: LEARNER
I20240502 14:17:41.846361 9538 consensus_queue.cc:260] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: NON_VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: true } }
I20240502 14:17:41.846817 9538 ts_tablet_manager.cc:1398] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:17:41.847609 7465 tablet_copy_service.cc:342] P 699831d1e03d4a109eb19f69687deac2: Request end of tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-41104faa3b9a403e8e2316f8c40d4036 received from {username='slave'} at 127.21.58.68:33195
I20240502 14:17:41.847790 7465 tablet_copy_service.cc:433] P 699831d1e03d4a109eb19f69687deac2: ending tablet copy session 1b8d0ccec3714c3694d8285fcceb4448-41104faa3b9a403e8e2316f8c40d4036 on tablet 41104faa3b9a403e8e2316f8c40d4036 with peer 1b8d0ccec3714c3694d8285fcceb4448
I20240502 14:17:41.975716 9517 ts_tablet_manager.cc:902] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: Initiating tablet copy from peer e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565)
I20240502 14:17:41.976810 9517 tablet_copy_client.cc:308] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.58.66:36565
I20240502 14:17:41.982383 7332 tablet_copy_service.cc:140] P e230e4b943ef49059c6a55ad1e667a02: Received BeginTabletCopySession request for tablet d0cd8c0d16f844edb70cec8092dd7f05 from peer 952df5ac1283460489e81cdb345007b7 ({username='slave'} at 127.21.58.69:49839)
I20240502 14:17:41.982539 7332 tablet_copy_service.cc:161] P e230e4b943ef49059c6a55ad1e667a02: Beginning new tablet copy session on tablet d0cd8c0d16f844edb70cec8092dd7f05 from peer 952df5ac1283460489e81cdb345007b7 at {username='slave'} at 127.21.58.69:49839: session id = 952df5ac1283460489e81cdb345007b7-d0cd8c0d16f844edb70cec8092dd7f05
I20240502 14:17:41.984452 7332 tablet_copy_source_session.cc:205] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:17:41.985569 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cd8c0d16f844edb70cec8092dd7f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:41.989917 9517 tablet_copy_client.cc:783] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 0 data blocks...
I20240502 14:17:41.990128 9517 tablet_copy_client.cc:647] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:41.991490 9517 tablet_copy_client.cc:523] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:17:41.993691 9517 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: Bootstrap starting.
I20240502 14:17:42.024710 9517 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:42.025115 9517 tablet_bootstrap.cc:492] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: Bootstrap complete.
I20240502 14:17:42.025332 9517 ts_tablet_manager.cc:1367] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20240502 14:17:42.026008 9517 raft_consensus.cc:348] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:42.026203 9517 raft_consensus.cc:729] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Initialized, Role: LEARNER
I20240502 14:17:42.026443 9517 consensus_queue.cc:260] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: NON_VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: true } }
I20240502 14:17:42.026878 9517 ts_tablet_manager.cc:1398] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:17:42.027828 7332 tablet_copy_service.cc:342] P e230e4b943ef49059c6a55ad1e667a02: Request end of tablet copy session 952df5ac1283460489e81cdb345007b7-d0cd8c0d16f844edb70cec8092dd7f05 received from {username='slave'} at 127.21.58.69:49839
I20240502 14:17:42.027966 7332 tablet_copy_service.cc:433] P e230e4b943ef49059c6a55ad1e667a02: ending tablet copy session 952df5ac1283460489e81cdb345007b7-d0cd8c0d16f844edb70cec8092dd7f05 on tablet d0cd8c0d16f844edb70cec8092dd7f05 with peer 952df5ac1283460489e81cdb345007b7
I20240502 14:17:42.121743 8110 raft_consensus.cc:1203] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:17:42.298034 8242 raft_consensus.cc:1203] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:17:42.546877 8110 raft_consensus.cc:1203] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:17:42.657160 9566 raft_consensus.cc:1050] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72: attempting to promote NON_VOTER 952df5ac1283460489e81cdb345007b7 to VOTER
I20240502 14:17:42.657990 9566 consensus_queue.cc:237] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:42.666985 7441 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:42.667887 8110 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:42.668005 7309 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:42.668373 9566 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 14:17:42.668730 9621 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:42.669042 9611 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:42.671923 9611 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.672605 7309 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.675195 7441 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.676173 8110 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.678637 7030 catalog_manager.cc:5484] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 reported cstate change: config changed from index 5 to 6, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.686549 7178 consensus_queue.cc:237] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:42.688099 8110 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:42.688259 7441 raft_consensus.cc:1261] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Refusing update from remote peer f7ad4b3a72f04f259caff4473296ae72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:42.688776 9611 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:42.689082 9621 consensus_queue.cc:1035] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:42.691548 7442 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.691702 8110 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.690984 9566 raft_consensus.cc:2932] T c2c230f5dbc14ee886256e6710c817b9 P f7ad4b3a72f04f259caff4473296ae72 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.696560 7026 catalog_manager.cc:5484] T c2c230f5dbc14ee886256e6710c817b9 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 6 to 7, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "f7ad4b3a72f04f259caff4473296ae72" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:42.698076 7019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2c230f5dbc14ee886256e6710c817b9 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:42.699628 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c230f5dbc14ee886256e6710c817b9 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:42.700258 9629 tablet_replica.cc:330] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:42.700464 9629 raft_consensus.cc:2229] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:42.700757 9629 raft_consensus.cc:2258] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:42.701490 9629 ts_tablet_manager.cc:1875] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:42.705930 9629 ts_tablet_manager.cc:1888] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:17:42.706032 9629 log.cc:1196] T c2c230f5dbc14ee886256e6710c817b9 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/c2c230f5dbc14ee886256e6710c817b9
I20240502 14:17:42.706638 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet c2c230f5dbc14ee886256e6710c817b9 (table rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]) successfully deleted
I20240502 14:17:42.803730 9584 raft_consensus.cc:1050] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2: attempting to promote NON_VOTER 1b8d0ccec3714c3694d8285fcceb4448 to VOTER
I20240502 14:17:42.804394 9584 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:42.806620 8242 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 LEARNER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:42.807303 8110 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:42.807323 7309 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:42.807359 9633 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:42.807940 9584 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:42.808241 9630 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:42.816133 9630 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:42.816933 8242 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:42.819937 8110 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:42.820796 7034 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:42.823522 7309 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:42.828238 7442 consensus_queue.cc:237] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } }
I20240502 14:17:42.829797 8242 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:42.830008 8110 raft_consensus.cc:1261] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Refusing update from remote peer 699831d1e03d4a109eb19f69687deac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:17:42.830376 9630 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:42.830785 9633 consensus_queue.cc:1035] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:42.832455 9630 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:42.833070 8242 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:42.833586 8110 raft_consensus.cc:2932] T 41104faa3b9a403e8e2316f8c40d4036 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } }
I20240502 14:17:42.835224 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41104faa3b9a403e8e2316f8c40d4036 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:42.836881 7035 catalog_manager.cc:5484] T 41104faa3b9a403e8e2316f8c40d4036 P 699831d1e03d4a109eb19f69687deac2 reported cstate change: config changed from index 9 to 10, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 1 leader_uuid: "699831d1e03d4a109eb19f69687deac2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "699831d1e03d4a109eb19f69687deac2" member_type: VOTER last_known_addr { host: "127.21.58.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:42.840444 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet 41104faa3b9a403e8e2316f8c40d4036 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:42.840786 9629 tablet_replica.cc:330] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:42.840960 9629 raft_consensus.cc:2229] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:42.841219 9629 raft_consensus.cc:2258] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:42.841969 9629 ts_tablet_manager.cc:1875] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:42.846035 9629 ts_tablet_manager.cc:1888] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:17:42.846148 9629 log.cc:1196] T 41104faa3b9a403e8e2316f8c40d4036 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/41104faa3b9a403e8e2316f8c40d4036
I20240502 14:17:42.846801 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet 41104faa3b9a403e8e2316f8c40d4036 (table rebalance_test_table_0 [id=68f336ae5ab04e7e9848e669f4b88a01]) successfully deleted
I20240502 14:17:43.140609 9601 raft_consensus.cc:1050] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: attempting to promote NON_VOTER 952df5ac1283460489e81cdb345007b7 to VOTER
I20240502 14:17:43.141479 9601 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:43.143656 8110 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 1 LEARNER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:43.144181 9659 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:43.144559 8242 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:43.144917 7178 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Refusing update from remote peer e230e4b943ef49059c6a55ad1e667a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:17:43.145175 9659 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:43.145498 9661 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:43.148236 9661 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:43.149003 8242 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:43.149122 7178 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:43.152844 8110 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:43.153223 7035 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 reported cstate change: config changed from index 8 to 9, 952df5ac1283460489e81cdb345007b7 (127.21.58.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e230e4b943ef49059c6a55ad1e667a02" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:43.317727 7309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05"
dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35788
I20240502 14:17:43.318001 7309 raft_consensus.cc:595] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:43.534746 7309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05"
dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35810
I20240502 14:17:43.534933 7309 raft_consensus.cc:595] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:17:43.676033 9687 raft_consensus.cc:979] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: : Instructing follower 952df5ac1283460489e81cdb345007b7 to start an election
I20240502 14:17:43.676225 9687 raft_consensus.cc:1067] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Signalling peer 952df5ac1283460489e81cdb345007b7 to start an election
I20240502 14:17:43.677100 8110 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05"
dest_uuid: "952df5ac1283460489e81cdb345007b7"
from {username='slave'} at 127.21.58.66:44543
I20240502 14:17:43.677348 8110 raft_consensus.cc:482] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:43.677462 8110 raft_consensus.cc:3037] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:43.679497 8110 raft_consensus.cc:504] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:43.681152 8110 leader_election.cc:288] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [CANDIDATE]: Term 2 election: Requested vote from peers e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565), f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021), 1b8d0ccec3714c3694d8285fcceb4448 (127.21.58.68:38889)
I20240502 14:17:43.681408 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05" candidate_uuid: "952df5ac1283460489e81cdb345007b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e230e4b943ef49059c6a55ad1e667a02"
I20240502 14:17:43.681622 7309 raft_consensus.cc:3032] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:43.681721 7309 raft_consensus.cc:729] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e230e4b943ef49059c6a55ad1e667a02, State: Running, Role: LEADER
I20240502 14:17:43.681749 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05" candidate_uuid: "952df5ac1283460489e81cdb345007b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "f7ad4b3a72f04f259caff4473296ae72"
I20240502 14:17:43.681947 7178 raft_consensus.cc:3037] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:43.682049 7309 consensus_queue.cc:260] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:43.682543 7309 raft_consensus.cc:3037] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:43.684762 7178 raft_consensus.cc:2454] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952df5ac1283460489e81cdb345007b7 in term 2.
I20240502 14:17:43.685695 7309 raft_consensus.cc:2454] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952df5ac1283460489e81cdb345007b7 in term 2.
I20240502 14:17:43.686131 8046 leader_election.cc:302] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 952df5ac1283460489e81cdb345007b7, e230e4b943ef49059c6a55ad1e667a02, f7ad4b3a72f04f259caff4473296ae72; no voters:
I20240502 14:17:43.687119 9690 raft_consensus.cc:2781] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:43.687695 9690 raft_consensus.cc:686] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 2 LEADER]: Becoming Leader. State: Replica: 952df5ac1283460489e81cdb345007b7, State: Running, Role: LEADER
I20240502 14:17:43.688230 9690 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:43.691705 8242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cd8c0d16f844edb70cec8092dd7f05" candidate_uuid: "952df5ac1283460489e81cdb345007b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1b8d0ccec3714c3694d8285fcceb4448"
I20240502 14:17:43.691996 8242 raft_consensus.cc:3037] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:43.691743 7026 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 reported cstate change: term changed from 1 to 2, leader changed from e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) to 952df5ac1283460489e81cdb345007b7 (127.21.58.69). New cstate: current_term: 2 leader_uuid: "952df5ac1283460489e81cdb345007b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:17:43.694347 8242 raft_consensus.cc:2454] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952df5ac1283460489e81cdb345007b7 in term 2.
I20240502 14:17:44.067559 7178 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Refusing update from remote peer 952df5ac1283460489e81cdb345007b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:44.068203 9690 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:44.093701 8242 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 FOLLOWER]: Refusing update from remote peer 952df5ac1283460489e81cdb345007b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:44.094274 9690 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:17:44.098560 7309 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Refusing update from remote peer 952df5ac1283460489e81cdb345007b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:17:44.099743 9708 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230e4b943ef49059c6a55ad1e667a02" member_type: VOTER last_known_addr { host: "127.21.58.66" port: 36565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240502 14:17:44.109057 8110 consensus_queue.cc:237] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } }
I20240502 14:17:44.110549 8242 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 FOLLOWER]: Refusing update from remote peer 952df5ac1283460489e81cdb345007b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:17:44.110743 7178 raft_consensus.cc:1261] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Refusing update from remote peer 952df5ac1283460489e81cdb345007b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:17:44.111291 9690 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:17:44.111678 9708 consensus_queue.cc:1035] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:17:44.113257 9690 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:44.113901 7178 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:44.114096 8242 raft_consensus.cc:2932] T d0cd8c0d16f844edb70cec8092dd7f05 P 1b8d0ccec3714c3694d8285fcceb4448 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:44.117333 7021 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0cd8c0d16f844edb70cec8092dd7f05 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:17:44.117614 7026 catalog_manager.cc:5484] T d0cd8c0d16f844edb70cec8092dd7f05 P f7ad4b3a72f04f259caff4473296ae72 reported cstate change: config changed from index 9 to 11, VOTER e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66) evicted. New cstate: current_term: 2 leader_uuid: "952df5ac1283460489e81cdb345007b7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f7ad4b3a72f04f259caff4473296ae72" member_type: VOTER last_known_addr { host: "127.21.58.65" port: 37021 } } peers { permanent_uuid: "1b8d0ccec3714c3694d8285fcceb4448" member_type: VOTER last_known_addr { host: "127.21.58.68" port: 38889 } attrs { promote: false } } peers { permanent_uuid: "952df5ac1283460489e81cdb345007b7" member_type: VOTER last_known_addr { host: "127.21.58.69" port: 34127 } attrs { promote: false } } }
I20240502 14:17:44.121378 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cd8c0d16f844edb70cec8092dd7f05 with delete_type TABLET_DATA_TOMBSTONED (TS e230e4b943ef49059c6a55ad1e667a02 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45870
I20240502 14:17:44.122051 9713 tablet_replica.cc:330] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: stopping tablet replica
I20240502 14:17:44.122260 9713 raft_consensus.cc:2229] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:44.122483 9713 raft_consensus.cc:2258] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:44.123425 9713 ts_tablet_manager.cc:1875] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:44.127513 9713 ts_tablet_manager.cc:1888] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 14:17:44.127619 9713 log.cc:1196] T d0cd8c0d16f844edb70cec8092dd7f05 P e230e4b943ef49059c6a55ad1e667a02: Deleting WAL directory at /tmp/dist-test-task3cgK4A/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714659268621181-21737-0/raft_consensus-itest-cluster/ts-1/wal/wals/d0cd8c0d16f844edb70cec8092dd7f05
I20240502 14:17:44.128249 7021 catalog_manager.cc:4830] TS e230e4b943ef49059c6a55ad1e667a02 (127.21.58.66:36565): tablet d0cd8c0d16f844edb70cec8092dd7f05 (table rebalance_test_table_2 [id=c34eddd4660047b59c6392b240407e0d]) successfully deleted
I20240502 14:17:44.459703 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:44.460444 8091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:44.468150 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:44.476205 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:44.486243 7292 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:44.634070 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:44.640388 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:44.646030 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:44.647485 8091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:44.662995 7292 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:44.816888 7292 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:44.817653 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:44.818859 8091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:44.819744 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:44.820356 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:44.992627 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:44.993561 8091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:44.995649 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:44.995786 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:45.007544 7292 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:45.167299 7157 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:17:45.169350 7422 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:17:45.170146 8222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:45.181001 7292 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:17:45.196527 8091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:17:45.394162 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 7095
I20240502 14:17:45.403105 7219 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:45.559635 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 7095
W20240502 14:17:45.587697 8044 connection.cc:535] server connection from 127.21.58.65:37999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:45.587710 8176 connection.cc:535] server connection from 127.21.58.65:45577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:45.588060 7377 connection.cc:535] server connection from 127.21.58.65:36815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:45.588096 8044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.58.65:37021 (error 108)
I20240502 14:17:45.588613 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 7228
W20240502 14:17:45.590283 8044 consensus_peers.cc:485] T d0cd8c0d16f844edb70cec8092dd7f05 P 952df5ac1283460489e81cdb345007b7 -> Peer f7ad4b3a72f04f259caff4473296ae72 (127.21.58.65:37021): Couldn't send request to peer f7ad4b3a72f04f259caff4473296ae72. Status: Network error: Client connection negotiation failed: client connection to 127.21.58.65:37021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:45.594522 7352 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:45.752056 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 7228
I20240502 14:17:45.780753 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 7360
I20240502 14:17:45.786751 7484 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:45.934602 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 7360
W20240502 14:17:45.961997 7019 connection.cc:535] server connection from 127.21.58.67:35761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:17:45.962383 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 8160
I20240502 14:17:45.968382 8282 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:46.101833 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 8160
I20240502 14:17:46.122561 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 8029
I20240502 14:17:46.128589 8150 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:46.260308 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 8029
I20240502 14:17:46.284936 21737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3cgK4A/build/asan/bin/kudu pid 7004
I20240502 14:17:46.286005 7066 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:17:46.441599 21737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cgK4A/build/asan/bin/kudu with pid 7004
2024-05-02T14:17:46Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (26804 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (51038 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (197858 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>
1 FAILED TEST